diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 248e38b61..b50b64cd6 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -1,58 +1,19 @@ .. _topics-broad-crawls: ============ -Broad Crawls +Broad crawls ============ -Scrapy defaults are optimized for crawling specific sites. These sites are -often handled by a single Scrapy spider, although this is not necessary or -required (for example, there are generic spiders that handle any given site -thrown at them). +While Scrapy is well suited for **broad crawls**, i.e. crawls that target many +websites, the default :ref:`settings ` are optimized for +crawls targetting a single website. -In addition to this "focused crawl", there is another common type of crawling -which covers a large (potentially unlimited) number of domains, and is only -limited by time or other arbitrary constraint, rather than stopping when the -domain was crawled to completion or when there are no more requests to perform. -These are called "broad crawls" and is the typical crawlers employed by search -engines. - -These are some common properties often found in broad crawls: - -* they crawl many domains (often, unbounded) instead of a specific set of sites - -* they don't necessarily crawl domains to completion, because it would be - impractical (or impossible) to do so, and instead limit the crawl by time or - number of pages crawled - -* they are simpler in logic (as opposed to very complex spiders with many - extraction rules) because data is often post-processed in a separate stage - -* they crawl many domains concurrently, which allows them to achieve faster - crawl speeds by not being limited by any particular site constraint (each site - is crawled slowly to respect politeness, but many sites are crawled in - parallel) - -As said above, Scrapy default settings are optimized for focused crawls, not -broad crawls. However, due to its asynchronous architecture, Scrapy is very -well suited for performing fast broad crawls. This page summarizes some things -you need to keep in mind when using Scrapy for doing broad crawls, along with -concrete suggestions of Scrapy settings to tune in order to achieve an -efficient broad crawl. +For broad crawls, consider these adjustments: .. _broad-crawls-scheduler-priority-queue: -Use the right :setting:`SCHEDULER_PRIORITY_QUEUE` -================================================= - -Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``. -It works best during single-domain crawl. It does not work well with crawling -many different domains in parallel - -To apply the recommended priority queue use: - -.. code-block:: python - - SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue" +- Set :setting:`SCHEDULER_PRIORITY_QUEUE` to + :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`. .. _broad-crawls-concurrency: diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a81832023..de7840878 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -127,10 +127,7 @@ Request objects body to bytes (if given as a string). :type encoding: str - :param priority: the priority of this request (defaults to ``0``). - The priority is used by the scheduler to define the order used to process - requests. Requests with a higher priority value will execute earlier. - Negative values are allowed in order to indicate relatively low-priority. + :param priority: sets :attr:`priority`, defaults to ``0``. :type priority: int :param dont_filter: sets :attr:`dont_filter`, defaults to ``False``. @@ -179,6 +176,8 @@ Request objects .. autoattribute:: errback + .. autoattribute:: priority + .. attribute:: Request.cb_kwargs A dictionary that contains arbitrary metadata for this request. Its contents diff --git a/docs/topics/scheduler.rst b/docs/topics/scheduler.rst index 57c24b76a..30ddda492 100644 --- a/docs/topics/scheduler.rst +++ b/docs/topics/scheduler.rst @@ -6,29 +6,50 @@ Scheduler .. module:: scrapy.core.scheduler -The scheduler component receives requests from the :ref:`engine ` -and stores them into persistent and/or non-persistent data structures. -It also gets those requests and feeds them back to the engine when it -asks for a next request to be downloaded. +The **scheduler** is a :ref:`component ` that stores pending +requests, drops unwanted requests, and determines in which order pending +requests are sent. + +It is set in the :setting:`SCHEDULER` setting. + +Pending requests may come from seeding (see :setting:`SEEDING_POLICY`), +spider callbacks (:attr:`Request.callback `), +:ref:`spider middlewares ` or :ref:`downloader +middlewares `. + +How requests are **stored** depends on the scheduler. The built-in scheduler, +:class:`~scrapy.core.scheduler.Scheduler`, can store requests in memory or +disk. Other schedulers may rely, for example, on frontier, queue, database or +storage services. + +Which requests are **dropped** also depends on the scheduler. It is recommended +for schedulers to use the configured :setting:`DUPEFILTER_CLASS` and take into +account :attr:`Request.dont_filter `, but +schedulers are free to follow their own criteria for dropping requests. + +How requests are **prioritized**, i.e. in which order they are sent, depends on +the scheduler as well. Schedulers may take into account :attr:`Request.priority +` and applicable built-in settings (e.g. +:setting:`SCHEDULER_PRIORITY_QUEUE`, :setting:`SCHEDULER_MEMORY_QUEUE`, +:setting:`SCHEDULER_DISK_QUEUE`), but schedulers may also ignore any of those +parameters at will. + +Built-in scheduler +================== + +.. autoclass:: Scheduler() -Overriding the default scheduler -================================ +Writing a scheduler +=================== -You can use your own custom scheduler class by supplying its full -Python path in the :setting:`SCHEDULER` setting. +.. tip:: Before writing a custom scheduler, see + :class:`~scrapy.core.scheduler.Scheduler` to learn how to customize the + default scheduler. - -Minimal scheduler interface -=========================== +Schedulers should subclass :class:`BaseScheduler` and implement its abstract +methods: .. autoclass:: BaseScheduler :members: - - -Default Scrapy scheduler -======================== - -.. autoclass:: Scheduler - :members: - :special-members: __len__ + :member-order: bysource diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 7aa340d8f..a1a4ab710 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1662,10 +1662,9 @@ the user agent to use in the robots.txt file. SCHEDULER --------- -Default: ``'scrapy.core.scheduler.Scheduler'`` +Default: :class:`~scrapy.core.scheduler.Scheduler` -The scheduler class to be used for crawling. -See the :ref:`topics-scheduler` topic for details. +See :ref:`scheduler `. .. setting:: SCHEDULER_DEBUG @@ -1710,14 +1709,19 @@ Type of in-memory queue used by scheduler. Other available type is: SCHEDULER_PRIORITY_QUEUE ------------------------ -Default: ``'scrapy.pqueues.ScrapyPriorityQueue'`` -Type of priority queue used by the scheduler. Another available type is -``scrapy.pqueues.DownloaderAwarePriorityQueue``. -``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than -``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different -domains in parallel. But currently ``scrapy.pqueues.DownloaderAwarePriorityQueue`` -does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`. +Default: :class:`~scrapy.pqueues.ScrapyPriorityQueue` + +Queue used by the :ref:`scheduler ` to sort scheduled +requests by :attr:`Request.priority `. + +Scheduled requests with the same priority are stored in nested queues, either +:setting:`SCHEDULER_MEMORY_QUEUE` or :setting:`SCHEDULER_DISK_QUEUE`. + +The following built-in priority queues are available: + +.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue +.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue .. setting:: SCRAPER_SLOT_MAX_ACTIVE_SIZE diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index 287b15c4b..efdbce5aa 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -50,130 +50,105 @@ class BaseSchedulerMeta(type): class BaseScheduler(metaclass=BaseSchedulerMeta): - """ - The scheduler component is responsible for storing requests received from - the engine, and feeding them back upon request (also to the engine). - - The original sources of said requests are: - - * Spider: ``yield_seeds`` method, requests created for URLs in the ``start_urls`` attribute, request callbacks - * Spider middleware: ``process_spider_output`` and ``process_spider_exception`` methods - * Downloader middleware: ``process_request``, ``process_response`` and ``process_exception`` methods - - The order in which the scheduler returns its stored requests (via the ``next_request`` method) - plays a great part in determining the order in which those requests are downloaded. - - The methods defined in this class constitute the minimal interface that the Scrapy engine will interact with. - """ - - @classmethod - def from_crawler(cls, crawler: Crawler) -> Self: - """ - Factory method which receives the current :class:`~scrapy.crawler.Crawler` object as argument. - """ - return cls() - - def open(self, spider: Spider) -> Deferred[None] | None: - """ - Called when the spider is opened by the engine. It receives the spider - instance as argument and it's useful to execute initialization code. - - :param spider: the spider object for the current crawl - :type spider: :class:`~scrapy.spiders.Spider` - """ - - def close(self, reason: str) -> Deferred[None] | None: - """ - Called when the spider is closed by the engine. It receives the reason why the crawl - finished as argument and it's useful to execute cleaning code. - - :param reason: a string which describes the reason why the spider was closed - :type reason: :class:`str` - """ - - @abstractmethod - def has_pending_requests(self) -> bool: - """ - ``True`` if the scheduler has enqueued requests, ``False`` otherwise - """ - raise NotImplementedError + """Base class for :ref:`schedulers `.""" @abstractmethod def enqueue_request(self, request: Request) -> bool: - """ - Process a request received by the engine. + """Store or drop *request*. - Return ``True`` if the request is stored correctly, ``False`` otherwise. + Return ``True`` if the request is stored or ``False`` if the request is + dropped, e.g. because it is deemed a duplicate of a previously-seen + request. - If ``False``, the engine will fire a ``request_dropped`` signal, and - will not make further attempts to schedule the request at a later time. - For reference, the default Scrapy scheduler returns ``False`` when the - request is rejected by the dupefilter. + Returning ``False`` triggers the :signal:`request_dropped` signal. """ raise NotImplementedError @abstractmethod def next_request(self) -> Request | None: - """ - Return the next :class:`~scrapy.Request` to be processed, or ``None`` - to indicate that there are no requests to be considered ready at the moment. + """Return the next :class:`~scrapy.Request` to send or ``None`` if + there are no requests to be sent. - Returning ``None`` implies that no request from the scheduler will be sent - to the downloader in the current reactor cycle. The engine will continue - calling ``next_request`` until ``has_pending_requests`` is ``False``. + .. note:: Returning ``None`` does not prevent future calls to this + method. See :meth:`has_pending_requests`. """ raise NotImplementedError + @abstractmethod + def has_pending_requests(self) -> bool: + """Return ``True`` if there are pending requests or ``False`` + otherwise. + + It is OK to return ``True`` even is the next call to + :meth:`next_request` returns ``None``. + + .. tip:: If you do this with the goal of feeding your crawl *start* + requests from a slow resource, like a network service, instead of a + custom scheduler, consider writing a :ref:`spider middleware + ` that implements + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`. + + .. warning:: The crawl will continue running as long as this method + returns ``True``. + """ + raise NotImplementedError + + def open(self, spider: Spider) -> Deferred[None] | None: + """Called after the spider opens. + + Useful for initialization code that needs to run later than the + ``__init__`` method, e.g. once seed iteration (see + :meth:`~scrapy.Spider.yield_seeds`) has started. + + May return a :class:`~twisted.internet.defer.Deferred`. + """ + + def close(self, reason: str) -> Deferred[None] | None: + """Called after the spider closes due to *reason* (see + :exc:`~scrapy.exceptions.CloseSpider`). + + Useful for cleanup code. + + May return a :class:`~twisted.internet.defer.Deferred`. + """ + class Scheduler(BaseScheduler): - """ - Default Scrapy scheduler. This implementation also handles duplication - filtering via the :setting:`dupefilter `. + """Default :ref:`scheduler `. - This scheduler stores requests into several priority queues (defined by the - :setting:`SCHEDULER_PRIORITY_QUEUE` setting). In turn, said priority queues - are backed by either memory or disk based queues (respectively defined by the - :setting:`SCHEDULER_MEMORY_QUEUE` and :setting:`SCHEDULER_DISK_QUEUE` settings). + Requests are stored in memory by default. Set :setting:`JOBDIR` to switch + to disk storage. - Request prioritization is almost entirely delegated to the priority queue. The only - prioritization performed by this scheduler is using the disk-based queue if present - (i.e. if the :setting:`JOBDIR` setting is defined) and falling back to the memory-based - queue if a serialization error occurs. If the disk queue is not present, the memory one - is used directly. + Requests are dropped if :attr:`~scrapy.Request.dont_filter` is ``False`` + and :setting:`DUPEFILTER_CLASS` flags them as duplicate requests. - :param dupefilter: An object responsible for checking and filtering duplicate requests. - The value for the :setting:`DUPEFILTER_CLASS` setting is used by default. - :type dupefilter: :class:`scrapy.dupefilters.BaseDupeFilter` instance or similar: - any class that implements the `BaseDupeFilter` interface + :setting:`SCHEDULER_PRIORITY_QUEUE` handles request prioritization. For + same-priority requests, their prioritization depends on + :setting:`SCHEDULER_MEMORY_QUEUE`, and also on + :setting:`SCHEDULER_DISK_QUEUE` if :setting:`JOBDIR` is set. - :param jobdir: The path of a directory to be used for persisting the crawl's state. - The value for the :setting:`JOBDIR` setting is used by default. - See :ref:`topics-jobs`. - :type jobdir: :class:`str` or ``None`` + If :setting:`JOBDIR` is set, :setting:`SCHEDULER_MEMORY_QUEUE` is used for + requests that cannot be serialized to disk. Memory requests always take + priority over disk requests. - :param dqclass: A class to be used as persistent request queue. - The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default. - :type dqclass: class + The following stats are generated: - :param mqclass: A class to be used as non-persistent request queue. - The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default. - :type mqclass: class + .. code-block:: none - :param logunser: A boolean that indicates whether or not unserializable requests should be logged. - The value for the :setting:`SCHEDULER_DEBUG` setting is used by default. - :type logunser: bool + scheduler/enqueued + scheduler/enqueued/memory + scheduler/enqueued/disk + scheduler/dequeued + scheduler/dequeued/memory + scheduler/dequeued/disk + scheduler/unserializable - :param stats: A stats collector object to record stats about the request scheduling process. - The value for the :setting:`STATS_CLASS` setting is used by default. - :type stats: :class:`scrapy.statscollectors.StatsCollector` instance or similar: - any class that implements the `StatsCollector` interface + If the value of the ``scheduler/unserializable`` stat is non-zero, consider + enabling :setting:`SCHEDULER_DEBUG` to log a warning messages with details + about the first unserializable request, to try and figure out how to make + it serializable. - :param pqclass: A class to be used as priority queue for requests. - The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default. - :type pqclass: class - - :param crawler: The crawler object corresponding to the current crawl. - :type crawler: :class:`scrapy.crawler.Crawler` + .. seealso:: :ref:`topics-jobs` """ def __init__( @@ -198,9 +173,6 @@ class Scheduler(BaseScheduler): @classmethod def from_crawler(cls, crawler: Crawler) -> Self: - """ - Factory method, initializes the scheduler with arguments taken from the crawl settings - """ dupefilter_cls = load_object(crawler.settings["DUPEFILTER_CLASS"]) return cls( dupefilter=build_from_crawler(dupefilter_cls, crawler), @@ -217,21 +189,12 @@ class Scheduler(BaseScheduler): return len(self) > 0 def open(self, spider: Spider) -> Deferred[None] | None: - """ - (1) initialize the memory queue - (2) initialize the disk queue if the ``jobdir`` attribute is a valid directory - (3) return the result of the dupefilter's ``open`` method - """ self.spider: Spider = spider self.mqs: ScrapyPriorityQueue = self._mq() self.dqs: ScrapyPriorityQueue | None = self._dq() if self.dqdir else None return self.df.open() def close(self, reason: str) -> Deferred[None] | None: - """ - (1) dump pending requests to disk if there is a disk queue - (2) return the result of the dupefilter's ``close`` method - """ if self.dqs is not None: state = self.dqs.close() assert isinstance(self.dqdir, str) @@ -239,15 +202,6 @@ class Scheduler(BaseScheduler): return self.df.close(reason) def enqueue_request(self, request: Request) -> bool: - """ - Unless the received request is filtered out by the Dupefilter, attempt to push - it into the disk queue, falling back to pushing it into the memory queue. - - Increment the appropriate stats, such as: ``scheduler/enqueued``, - ``scheduler/enqueued/disk``, ``scheduler/enqueued/memory``. - - Return ``True`` if the request was stored successfully, ``False`` otherwise. - """ if not request.dont_filter and self.df.request_seen(request): self.df.log(request, self.spider) return False @@ -262,14 +216,6 @@ class Scheduler(BaseScheduler): return True def next_request(self) -> Request | None: - """ - Return a :class:`~scrapy.Request` object from the memory queue, - falling back to the disk queue if the memory queue is empty. - Return ``None`` if there are no more enqueued requests. - - Increment the appropriate stats, such as: ``scheduler/dequeued``, - ``scheduler/dequeued/disk``, ``scheduler/dequeued/memory``. - """ request: Request | None = self.mqs.pop() assert self.stats is not None if request is not None: @@ -283,9 +229,6 @@ class Scheduler(BaseScheduler): return request def __len__(self) -> int: - """ - Return the total amount of enqueued requests - """ return len(self.dqs) + len(self.mqs) if self.dqs is not None else len(self.mqs) def _dqpush(self, request: Request) -> bool: diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.py index 0d2b3aa99..62b9697f1 100644 --- a/scrapy/http/request/__init__.py +++ b/scrapy/http/request/__init__.py @@ -130,6 +130,14 @@ class Request(object_ref): self._set_body(body) if not isinstance(priority, int): raise TypeError(f"Request priority not an integer: {priority!r}") + + #: Value that the :ref:`scheduler ` may use for + #: request prioritization. + #: + #: Built-in schedulers prioritize requests with a higher priority + #: value. + #: + #: Negative values are allowed. self.priority: int = priority if not (callable(callback) or callback is None): diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py index 324a9b955..5aba0bf25 100644 --- a/scrapy/pqueues.py +++ b/scrapy/pqueues.py @@ -50,9 +50,9 @@ class QueueProtocol(Protocol): class ScrapyPriorityQueue: - """A priority queue implemented using multiple internal queues (typically, - FIFO queues). It uses one internal queue for each priority value. The internal - queue must implement the following methods: + """Default scheduler priority queue (:setting:`SCHEDULER_PRIORITY_QUEUE`). + + The internal queue must implement the following methods: * push(obj) * pop() @@ -185,6 +185,13 @@ class DownloaderAwarePriorityQueue: """PriorityQueue which takes Downloader activity into account: domains (slots) with the least amount of active downloads are dequeued first. + + Another available type is + ``scrapy.pqueues.DownloaderAwarePriorityQueue``. + ``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than + ``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different + domains in parallel. But currently ``scrapy.pqueues.DownloaderAwarePriorityQueue`` + does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`. """ @classmethod