diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index 50bcaa6d6..2a976e91d 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -17,15 +17,21 @@ facilities: * an extension that keeps some spider state (key/value pairs) persistent between batches +.. _job-dir: + Job directory ============= -To enable persistence support you just need to define a *job directory* through -the ``JOBDIR`` setting. This directory will be for storing all required data to -keep the state of a single job (i.e. a spider run). It's important to note that -this directory must not be shared by different spiders, or even different -jobs/runs of the same spider, as it's meant to be used for storing the state of -a *single* job. +To enable persistence support, define a *job directory* through the +:setting:`JOBDIR` setting. + +The job directory will store all required data to keep the state of a *single* +job (i.e. a spider run), so that if stopped cleanly, it can be resumed later. + +.. warning:: This directory must *not* be shared by different spiders, or even + different jobs of the same spider. + +See also :ref:`job-dir-contents`. How to use it ============= @@ -65,6 +71,14 @@ Persistence gotchas There are a few things to keep in mind if you want to be able to use the Scrapy persistence support: +Pause limitations +----------------- + +Job pausing and resuming is only supported when the spider is paused by +stopping it cleanly. Forced, sudden or otherwise unclean shutdown can lead to +data corruption in the job directory, which may prevent the spider from +resuming correctly. + Cookies expiration ------------------ @@ -72,7 +86,6 @@ Cookies may expire. So, if you don't resume your spider quickly the requests scheduled may no longer work. This won't be an issue if your spider doesn't rely on cookies. - .. _request-serialization: Request serialization @@ -86,3 +99,52 @@ running :class:`~scrapy.Spider` class. If you wish to log the requests that couldn't be serialized, you can set the :setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page. It is ``False`` by default. + +.. _job-dir-contents: + +Job directory contents +====================== + +The contents of a job directory depend on the components used during the job. +Components known to write in the job directory include the :ref:`scheduler +` and the :class:`~scrapy.extensions.spiderstate.SpiderState` +extension. See the reference documentation of the corresponding components for +details. + +For example, with default settings, the job directory may look like this: + +.. code-block:: none + + ├── requests.queue + | ├── active.json + | └── {hostname}-{hash} + | └── {priority}{s?} + | ├── q{00000} + | └── info.json + ├── requests.seen + └── spider.state + +Where: + +- :class:`~scrapy.core.scheduler.Scheduler` creates the ``requests.queue/`` + directory and the ``active.json`` file, the latter containing the state + data returned by :meth:`DownloaderAwarePriorityQueue.close() + ` the last time the job + was paused. + +- :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` creates the + ``{hostname}-{hash}`` directories. + +- :class:`~scrapy.pqueues.ScrapyPriorityQueue` creates the ``{priority}{s?}`` + directories. + +- :class:`scrapy.squeues.PickleLifoDiskQueue`, a subclass of + :class:`queuelib.LifoDiskQueue` that uses :mod:`pickle` to serialize + :class:`dict` representations of :class:`scrapy.Request` objects, creates + the ``info.json`` and ``q{00000}`` files. + +- :class:`~scrapy.dupefilters.RFPDupeFilter` creates the ``requests.seen`` + file. + +- :class:`~scrapy.extensions.spiderstate.SpiderState` creates the + ``spider.state`` file. diff --git a/docs/topics/scheduler.rst b/docs/topics/scheduler.rst index b6e54ebd7..b79d6de1f 100644 --- a/docs/topics/scheduler.rst +++ b/docs/topics/scheduler.rst @@ -32,3 +32,10 @@ Default scheduler .. autoclass:: Scheduler() :members: :special-members: __init__, __len__ + + +Priority queues +=============== + +.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue +.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 5357fa72b..5b4a9b5f9 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1702,10 +1702,10 @@ the user agent to use in the robots.txt file. SCHEDULER --------- -Default: ``'scrapy.core.scheduler.Scheduler'`` +Default: :class:`~scrapy.core.scheduler.Scheduler` -The scheduler class to be used for crawling. -See the :ref:`topics-scheduler` topic for details. +The scheduler class to be used for crawling. See :ref:`topics-scheduler` for +details. .. setting:: SCHEDULER_DEBUG @@ -1755,12 +1755,14 @@ Type of in-memory queue used by the scheduler. Other available type is: SCHEDULER_PRIORITY_QUEUE ------------------------ -Default: ``'scrapy.pqueues.DownloaderAwarePriorityQueue'`` +Default: :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` -Type of priority queue used by the scheduler. Another available type is -``scrapy.pqueues.ScrapyPriorityQueue``. -``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than -``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different +Type of priority queue used by the scheduler. + +Another available type is :class:`~scrapy.pqueues.ScrapyPriorityQueue`. + +:class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` works better than +:class:`~scrapy.pqueues.ScrapyPriorityQueue` when you crawl many different domains in parallel. diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index 491e7a8a6..5a0aa2197 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -4,7 +4,7 @@ import json import logging from abc import abstractmethod from pathlib import Path -from typing import TYPE_CHECKING, Any, cast +from typing import TYPE_CHECKING, Any from warnings import warn # working around https://github.com/sphinx-doc/sphinx/issues/10400 @@ -128,7 +128,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta): class Scheduler(BaseScheduler): - """Default scheduler. + r"""Default scheduler. Requests are stored into priority queues (:setting:`SCHEDULER_PRIORITY_QUEUE`) that sort requests by @@ -190,7 +190,8 @@ class Scheduler(BaseScheduler): following :ref:`settings `: | :setting:`DEPTH_PRIORITY` = ``1`` - | :setting:`SCHEDULER_DISK_QUEUE` = ``"scrapy.squeues.PickleFifoDiskQueue"`` + | :setting:`SCHEDULER_DISK_QUEUE` = + ``"scrapy.squeues.PickleFifoDiskQueue"`` | :setting:`SCHEDULER_MEMORY_QUEUE` = ``"scrapy.squeues.FifoMemoryQueue"`` .. _BFO order: https://en.wikipedia.org/wiki/Breadth-first_search @@ -219,6 +220,37 @@ class Scheduler(BaseScheduler): order. Lowering those settings to ``1`` enforces the desired order except for the very first request, but it significantly slows down the crawl as a whole. + + Job directory contents + ====================== + + .. warning:: The files that this class generates in the :ref:`job directory + ` are an implementation detail, and may change without a + warning in a future version of Scrapy. Do not rely on the following + information for anything other than debugging purposes. + + When using :setting:`JOBDIR`, this scheduler class: + + - Creates a directory named ``requests.queue`` inside the :ref:`job + directory `, meant to keep track of all requests stored in + the scheduler (i.e. not downloaded yet). + + - Generates inside that directory an ``active.json`` file with a JSON + representation of the state (``startprios``) of + :setting:`SCHEDULER_PRIORITY_QUEUE`. + + The file is generated whenever the job stops (cleanly) and is loaded + when resuming the job. + + - Instantiates the configured :setting:`SCHEDULER_PRIORITY_QUEUE` with + ``requests.queue/`` as persistence directory (*key*) and + :setting:`SCHEDULER_DISK_QUEUE` as *downstream_queue_cls*. The priority + queue may create additional files and directories inside that + directory, directly or though instances of + :setting:`SCHEDULER_DISK_QUEUE`. + + This scheduler class also uses the configured :setting:`DUPEFILTER_CLASS`, + which may also write data inside the job directory. """ @classmethod @@ -486,13 +518,13 @@ class Scheduler(BaseScheduler): return str(dqdir) return None - def _read_dqs_state(self, dqdir: str) -> list[int]: + def _read_dqs_state(self, dqdir: str) -> Any: path = Path(dqdir, "active.json") if not path.exists(): return [] with path.open(encoding="utf-8") as f: - return cast("list[int]", json.load(f)) + return json.load(f) - def _write_dqs_state(self, dqdir: str, state: list[int]) -> None: + def _write_dqs_state(self, dqdir: str, state: Any) -> None: with Path(dqdir, "active.json").open("w", encoding="utf-8") as f: json.dump(state, f) diff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.py index f0a6988c8..36fb0f97d 100644 --- a/scrapy/dupefilters.py +++ b/scrapy/dupefilters.py @@ -55,6 +55,18 @@ class RFPDupeFilter(BaseDupeFilter): filters out requests with the canonical (:func:`w3lib.url.canonicalize_url`) :attr:`~scrapy.http.Request.url`, :attr:`~scrapy.http.Request.method` and :attr:`~scrapy.http.Request.body`. + + Job directory contents + ====================== + + .. warning:: The files that this class generates in the :ref:`job directory + ` are an implementation detail, and may change without a + warning in a future version of Scrapy. Do not rely on the following + information for anything other than debugging purposes. + + When using :setting:`JOBDIR`, seen fingerprints are tracked in a file named + ``requests.seen`` in the :ref:`job directory `, which contains 1 + request fingerprint per line. """ def __init__( diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py index 42c53a527..ed57091ba 100644 --- a/scrapy/pqueues.py +++ b/scrapy/pqueues.py @@ -51,16 +51,17 @@ class QueueProtocol(Protocol): class ScrapyPriorityQueue: """A priority queue implemented using multiple internal queues (typically, - FIFO queues). It uses one internal queue for each priority value. The internal - queue must implement the following methods: + FIFO queues). It uses one internal queue for each priority value. The + internal queue must implement the following methods: * push(obj) * pop() * close() * __len__() - Optionally, the queue could provide a ``peek`` method, that should return the - next object to be returned by ``pop``, but without removing it from the queue. + Optionally, the queue could provide a ``peek`` method, that should return + the next object to be returned by ``pop``, but without removing it from the + queue. ``__init__`` method of ScrapyPriorityQueue receives a downstream_queue_cls argument, which is a class used to instantiate a new (internal) queue when @@ -72,6 +73,28 @@ class ScrapyPriorityQueue: startprios is a sequence of priorities to start with. If the queue was previously closed leaving some priority buckets non-empty, those priorities should be passed in startprios. + + Disk persistence + ================ + + .. warning:: The files that this class generates on disk are an + implementation detail, and may change without a warning in a future + version of Scrapy. Do not rely on the following information for + anything other than debugging purposes. + + When a component instantiates this class with a non-empty *key* argument, + *key* is used as a persistence directory. + + For every request enqueued, this class checks: + + - Whether the request is a :ref:`start request ` or not. + + - The :data:`~scrapy.Request.priority` of the request. + + For each combination of the above seen, this class creates an instance of + *downstream_queue_cls* with *key* set to a subdirectory of the persistence + directory, named as the request priority (e.g. ``1``), with an ``s`` suffix + in case of a start request (e.g. ``1s``). """ @classmethod @@ -255,6 +278,26 @@ class DownloaderAwarePriorityQueue: """PriorityQueue which takes Downloader activity into account: domains (slots) with the least amount of active downloads are dequeued first. + + Disk persistence + ================ + + .. warning:: The files that this class generates on disk are an + implementation detail, and may change without a warning in a future + version of Scrapy. Do not rely on the following information for + anything other than debugging purposes. + + When a component instantiates this class with a non-empty *key* argument, + *key* is used as a persistence directory, and inside that directory this + class creates a subdirectory per download slot (domain). + + Those subdirectories are named after the corresponding download slot, with + path-unsafe characters replaced by underscores and an MD5 hash suffix to + avoid collisions. + + For each download slot, this class creates an instance of + :class:`ScrapyPriorityQueue` with the download slot subdirectory as *key* + and its own *downstream_queue_cls*. """ @classmethod