mirror of https://github.com/scrapy/scrapy.git
WIP documentation improvements
This commit is contained in:
Adrián Chaves
parent
ba88283236
commit
65f3f0d208
|
|
@ -1,58 +1,19 @@
|
|||
.. _topics-broad-crawls:
|
||||
|
||||
============
|
||||
Broad Crawls
|
||||
Broad crawls
|
||||
============
|
||||
|
||||
Scrapy defaults are optimized for crawling specific sites. These sites are
|
||||
often handled by a single Scrapy spider, although this is not necessary or
|
||||
required (for example, there are generic spiders that handle any given site
|
||||
thrown at them).
|
||||
While Scrapy is well suited for **broad crawls**, i.e. crawls that target many
|
||||
websites, the default :ref:`settings <topics-settings>` are optimized for
|
||||
crawls targetting a single website.
|
||||
|
||||
In addition to this "focused crawl", there is another common type of crawling
|
||||
which covers a large (potentially unlimited) number of domains, and is only
|
||||
limited by time or other arbitrary constraint, rather than stopping when the
|
||||
domain was crawled to completion or when there are no more requests to perform.
|
||||
These are called "broad crawls" and is the typical crawlers employed by search
|
||||
engines.
|
||||
|
||||
These are some common properties often found in broad crawls:
|
||||
|
||||
* they crawl many domains (often, unbounded) instead of a specific set of sites
|
||||
|
||||
* they don't necessarily crawl domains to completion, because it would be
|
||||
impractical (or impossible) to do so, and instead limit the crawl by time or
|
||||
number of pages crawled
|
||||
|
||||
* they are simpler in logic (as opposed to very complex spiders with many
|
||||
extraction rules) because data is often post-processed in a separate stage
|
||||
|
||||
* they crawl many domains concurrently, which allows them to achieve faster
|
||||
crawl speeds by not being limited by any particular site constraint (each site
|
||||
is crawled slowly to respect politeness, but many sites are crawled in
|
||||
parallel)
|
||||
|
||||
As said above, Scrapy default settings are optimized for focused crawls, not
|
||||
broad crawls. However, due to its asynchronous architecture, Scrapy is very
|
||||
well suited for performing fast broad crawls. This page summarizes some things
|
||||
you need to keep in mind when using Scrapy for doing broad crawls, along with
|
||||
concrete suggestions of Scrapy settings to tune in order to achieve an
|
||||
efficient broad crawl.
|
||||
For broad crawls, consider these adjustments:
|
||||
|
||||
.. _broad-crawls-scheduler-priority-queue:
|
||||
|
||||
Use the right :setting:`SCHEDULER_PRIORITY_QUEUE`
|
||||
=================================================
|
||||
|
||||
Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``.
|
||||
It works best during single-domain crawl. It does not work well with crawling
|
||||
many different domains in parallel
|
||||
|
||||
To apply the recommended priority queue use:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue"
|
||||
- Set :setting:`SCHEDULER_PRIORITY_QUEUE` to
|
||||
:class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.
|
||||
|
||||
.. _broad-crawls-concurrency:
|
||||
|
||||
|
|
|
|||
|
|
@ -127,10 +127,7 @@ Request objects
|
|||
body to bytes (if given as a string).
|
||||
:type encoding: str
|
||||
|
||||
:param priority: the priority of this request (defaults to ``0``).
|
||||
The priority is used by the scheduler to define the order used to process
|
||||
requests. Requests with a higher priority value will execute earlier.
|
||||
Negative values are allowed in order to indicate relatively low-priority.
|
||||
:param priority: sets :attr:`priority`, defaults to ``0``.
|
||||
:type priority: int
|
||||
|
||||
:param dont_filter: sets :attr:`dont_filter`, defaults to ``False``.
|
||||
|
|
@ -179,6 +176,8 @@ Request objects
|
|||
|
||||
.. autoattribute:: errback
|
||||
|
||||
.. autoattribute:: priority
|
||||
|
||||
.. attribute:: Request.cb_kwargs
|
||||
|
||||
A dictionary that contains arbitrary metadata for this request. Its contents
|
||||
|
|
|
|||
|
|
@ -6,29 +6,50 @@ Scheduler
|
|||
|
||||
.. module:: scrapy.core.scheduler
|
||||
|
||||
The scheduler component receives requests from the :ref:`engine <component-engine>`
|
||||
and stores them into persistent and/or non-persistent data structures.
|
||||
It also gets those requests and feeds them back to the engine when it
|
||||
asks for a next request to be downloaded.
|
||||
The **scheduler** is a :ref:`component <topics-components>` that stores pending
|
||||
requests, drops unwanted requests, and determines in which order pending
|
||||
requests are sent.
|
||||
|
||||
It is set in the :setting:`SCHEDULER` setting.
|
||||
|
||||
Pending requests may come from seeding (see :setting:`SEEDING_POLICY`),
|
||||
spider callbacks (:attr:`Request.callback <scrapy.Request.callback>`),
|
||||
:ref:`spider middlewares <topics-spider-middleware>` or :ref:`downloader
|
||||
middlewares <topics-downloader-middleware>`.
|
||||
|
||||
How requests are **stored** depends on the scheduler. The built-in scheduler,
|
||||
:class:`~scrapy.core.scheduler.Scheduler`, can store requests in memory or
|
||||
disk. Other schedulers may rely, for example, on frontier, queue, database or
|
||||
storage services.
|
||||
|
||||
Which requests are **dropped** also depends on the scheduler. It is recommended
|
||||
for schedulers to use the configured :setting:`DUPEFILTER_CLASS` and take into
|
||||
account :attr:`Request.dont_filter <scrapy.Request.dont_filter>`, but
|
||||
schedulers are free to follow their own criteria for dropping requests.
|
||||
|
||||
How requests are **prioritized**, i.e. in which order they are sent, depends on
|
||||
the scheduler as well. Schedulers may take into account :attr:`Request.priority
|
||||
<scrapy.Request.priority>` and applicable built-in settings (e.g.
|
||||
:setting:`SCHEDULER_PRIORITY_QUEUE`, :setting:`SCHEDULER_MEMORY_QUEUE`,
|
||||
:setting:`SCHEDULER_DISK_QUEUE`), but schedulers may also ignore any of those
|
||||
parameters at will.
|
||||
|
||||
Built-in scheduler
|
||||
==================
|
||||
|
||||
.. autoclass:: Scheduler()
|
||||
|
||||
|
||||
Overriding the default scheduler
|
||||
================================
|
||||
Writing a scheduler
|
||||
===================
|
||||
|
||||
You can use your own custom scheduler class by supplying its full
|
||||
Python path in the :setting:`SCHEDULER` setting.
|
||||
.. tip:: Before writing a custom scheduler, see
|
||||
:class:`~scrapy.core.scheduler.Scheduler` to learn how to customize the
|
||||
default scheduler.
|
||||
|
||||
|
||||
Minimal scheduler interface
|
||||
===========================
|
||||
Schedulers should subclass :class:`BaseScheduler` and implement its abstract
|
||||
methods:
|
||||
|
||||
.. autoclass:: BaseScheduler
|
||||
:members:
|
||||
|
||||
|
||||
Default Scrapy scheduler
|
||||
========================
|
||||
|
||||
.. autoclass:: Scheduler
|
||||
:members:
|
||||
:special-members: __len__
|
||||
:member-order: bysource
|
||||
|
|
|
|||
|
|
@ -1662,10 +1662,9 @@ the user agent to use in the robots.txt file.
|
|||
SCHEDULER
|
||||
---------
|
||||
|
||||
Default: ``'scrapy.core.scheduler.Scheduler'``
|
||||
Default: :class:`~scrapy.core.scheduler.Scheduler`
|
||||
|
||||
The scheduler class to be used for crawling.
|
||||
See the :ref:`topics-scheduler` topic for details.
|
||||
See :ref:`scheduler <topics-scheduler>`.
|
||||
|
||||
.. setting:: SCHEDULER_DEBUG
|
||||
|
||||
|
|
@ -1710,14 +1709,19 @@ Type of in-memory queue used by scheduler. Other available type is:
|
|||
|
||||
SCHEDULER_PRIORITY_QUEUE
|
||||
------------------------
|
||||
Default: ``'scrapy.pqueues.ScrapyPriorityQueue'``
|
||||
|
||||
Type of priority queue used by the scheduler. Another available type is
|
||||
``scrapy.pqueues.DownloaderAwarePriorityQueue``.
|
||||
``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than
|
||||
``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different
|
||||
domains in parallel. But currently ``scrapy.pqueues.DownloaderAwarePriorityQueue``
|
||||
does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`.
|
||||
Default: :class:`~scrapy.pqueues.ScrapyPriorityQueue`
|
||||
|
||||
Queue used by the :ref:`scheduler <topics-scheduler>` to sort scheduled
|
||||
requests by :attr:`Request.priority <scrapy.Request.priority>`.
|
||||
|
||||
Scheduled requests with the same priority are stored in nested queues, either
|
||||
:setting:`SCHEDULER_MEMORY_QUEUE` or :setting:`SCHEDULER_DISK_QUEUE`.
|
||||
|
||||
The following built-in priority queues are available:
|
||||
|
||||
.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue
|
||||
.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue
|
||||
|
||||
.. setting:: SCRAPER_SLOT_MAX_ACTIVE_SIZE
|
||||
|
||||
|
|
|
|||
|
|
@ -50,130 +50,105 @@ class BaseSchedulerMeta(type):
|
|||
|
||||
|
||||
class BaseScheduler(metaclass=BaseSchedulerMeta):
|
||||
"""
|
||||
The scheduler component is responsible for storing requests received from
|
||||
the engine, and feeding them back upon request (also to the engine).
|
||||
|
||||
The original sources of said requests are:
|
||||
|
||||
* Spider: ``yield_seeds`` method, requests created for URLs in the ``start_urls`` attribute, request callbacks
|
||||
* Spider middleware: ``process_spider_output`` and ``process_spider_exception`` methods
|
||||
* Downloader middleware: ``process_request``, ``process_response`` and ``process_exception`` methods
|
||||
|
||||
The order in which the scheduler returns its stored requests (via the ``next_request`` method)
|
||||
plays a great part in determining the order in which those requests are downloaded.
|
||||
|
||||
The methods defined in this class constitute the minimal interface that the Scrapy engine will interact with.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler: Crawler) -> Self:
|
||||
"""
|
||||
Factory method which receives the current :class:`~scrapy.crawler.Crawler` object as argument.
|
||||
"""
|
||||
return cls()
|
||||
|
||||
def open(self, spider: Spider) -> Deferred[None] | None:
|
||||
"""
|
||||
Called when the spider is opened by the engine. It receives the spider
|
||||
instance as argument and it's useful to execute initialization code.
|
||||
|
||||
:param spider: the spider object for the current crawl
|
||||
:type spider: :class:`~scrapy.spiders.Spider`
|
||||
"""
|
||||
|
||||
def close(self, reason: str) -> Deferred[None] | None:
|
||||
"""
|
||||
Called when the spider is closed by the engine. It receives the reason why the crawl
|
||||
finished as argument and it's useful to execute cleaning code.
|
||||
|
||||
:param reason: a string which describes the reason why the spider was closed
|
||||
:type reason: :class:`str`
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def has_pending_requests(self) -> bool:
|
||||
"""
|
||||
``True`` if the scheduler has enqueued requests, ``False`` otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
"""Base class for :ref:`schedulers <topics-scheduler>`."""
|
||||
|
||||
@abstractmethod
|
||||
def enqueue_request(self, request: Request) -> bool:
|
||||
"""
|
||||
Process a request received by the engine.
|
||||
"""Store or drop *request*.
|
||||
|
||||
Return ``True`` if the request is stored correctly, ``False`` otherwise.
|
||||
Return ``True`` if the request is stored or ``False`` if the request is
|
||||
dropped, e.g. because it is deemed a duplicate of a previously-seen
|
||||
request.
|
||||
|
||||
If ``False``, the engine will fire a ``request_dropped`` signal, and
|
||||
will not make further attempts to schedule the request at a later time.
|
||||
For reference, the default Scrapy scheduler returns ``False`` when the
|
||||
request is rejected by the dupefilter.
|
||||
Returning ``False`` triggers the :signal:`request_dropped` signal.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
def next_request(self) -> Request | None:
|
||||
"""
|
||||
Return the next :class:`~scrapy.Request` to be processed, or ``None``
|
||||
to indicate that there are no requests to be considered ready at the moment.
|
||||
"""Return the next :class:`~scrapy.Request` to send or ``None`` if
|
||||
there are no requests to be sent.
|
||||
|
||||
Returning ``None`` implies that no request from the scheduler will be sent
|
||||
to the downloader in the current reactor cycle. The engine will continue
|
||||
calling ``next_request`` until ``has_pending_requests`` is ``False``.
|
||||
.. note:: Returning ``None`` does not prevent future calls to this
|
||||
method. See :meth:`has_pending_requests`.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
def has_pending_requests(self) -> bool:
|
||||
"""Return ``True`` if there are pending requests or ``False``
|
||||
otherwise.
|
||||
|
||||
It is OK to return ``True`` even is the next call to
|
||||
:meth:`next_request` returns ``None``.
|
||||
|
||||
.. tip:: If you do this with the goal of feeding your crawl *start*
|
||||
requests from a slow resource, like a network service, instead of a
|
||||
custom scheduler, consider writing a :ref:`spider middleware
|
||||
<topics-spider-middleware>` that implements
|
||||
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`.
|
||||
|
||||
.. warning:: The crawl will continue running as long as this method
|
||||
returns ``True``.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def open(self, spider: Spider) -> Deferred[None] | None:
|
||||
"""Called after the spider opens.
|
||||
|
||||
Useful for initialization code that needs to run later than the
|
||||
``__init__`` method, e.g. once seed iteration (see
|
||||
:meth:`~scrapy.Spider.yield_seeds`) has started.
|
||||
|
||||
May return a :class:`~twisted.internet.defer.Deferred`.
|
||||
"""
|
||||
|
||||
def close(self, reason: str) -> Deferred[None] | None:
|
||||
"""Called after the spider closes due to *reason* (see
|
||||
:exc:`~scrapy.exceptions.CloseSpider`).
|
||||
|
||||
Useful for cleanup code.
|
||||
|
||||
May return a :class:`~twisted.internet.defer.Deferred`.
|
||||
"""
|
||||
|
||||
|
||||
class Scheduler(BaseScheduler):
|
||||
"""
|
||||
Default Scrapy scheduler. This implementation also handles duplication
|
||||
filtering via the :setting:`dupefilter <DUPEFILTER_CLASS>`.
|
||||
"""Default :ref:`scheduler <topics-scheduler>`.
|
||||
|
||||
This scheduler stores requests into several priority queues (defined by the
|
||||
:setting:`SCHEDULER_PRIORITY_QUEUE` setting). In turn, said priority queues
|
||||
are backed by either memory or disk based queues (respectively defined by the
|
||||
:setting:`SCHEDULER_MEMORY_QUEUE` and :setting:`SCHEDULER_DISK_QUEUE` settings).
|
||||
Requests are stored in memory by default. Set :setting:`JOBDIR` to switch
|
||||
to disk storage.
|
||||
|
||||
Request prioritization is almost entirely delegated to the priority queue. The only
|
||||
prioritization performed by this scheduler is using the disk-based queue if present
|
||||
(i.e. if the :setting:`JOBDIR` setting is defined) and falling back to the memory-based
|
||||
queue if a serialization error occurs. If the disk queue is not present, the memory one
|
||||
is used directly.
|
||||
Requests are dropped if :attr:`~scrapy.Request.dont_filter` is ``False``
|
||||
and :setting:`DUPEFILTER_CLASS` flags them as duplicate requests.
|
||||
|
||||
:param dupefilter: An object responsible for checking and filtering duplicate requests.
|
||||
The value for the :setting:`DUPEFILTER_CLASS` setting is used by default.
|
||||
:type dupefilter: :class:`scrapy.dupefilters.BaseDupeFilter` instance or similar:
|
||||
any class that implements the `BaseDupeFilter` interface
|
||||
:setting:`SCHEDULER_PRIORITY_QUEUE` handles request prioritization. For
|
||||
same-priority requests, their prioritization depends on
|
||||
:setting:`SCHEDULER_MEMORY_QUEUE`, and also on
|
||||
:setting:`SCHEDULER_DISK_QUEUE` if :setting:`JOBDIR` is set.
|
||||
|
||||
:param jobdir: The path of a directory to be used for persisting the crawl's state.
|
||||
The value for the :setting:`JOBDIR` setting is used by default.
|
||||
See :ref:`topics-jobs`.
|
||||
:type jobdir: :class:`str` or ``None``
|
||||
If :setting:`JOBDIR` is set, :setting:`SCHEDULER_MEMORY_QUEUE` is used for
|
||||
requests that cannot be serialized to disk. Memory requests always take
|
||||
priority over disk requests.
|
||||
|
||||
:param dqclass: A class to be used as persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default.
|
||||
:type dqclass: class
|
||||
The following stats are generated:
|
||||
|
||||
:param mqclass: A class to be used as non-persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default.
|
||||
:type mqclass: class
|
||||
.. code-block:: none
|
||||
|
||||
:param logunser: A boolean that indicates whether or not unserializable requests should be logged.
|
||||
The value for the :setting:`SCHEDULER_DEBUG` setting is used by default.
|
||||
:type logunser: bool
|
||||
scheduler/enqueued
|
||||
scheduler/enqueued/memory
|
||||
scheduler/enqueued/disk
|
||||
scheduler/dequeued
|
||||
scheduler/dequeued/memory
|
||||
scheduler/dequeued/disk
|
||||
scheduler/unserializable
|
||||
|
||||
:param stats: A stats collector object to record stats about the request scheduling process.
|
||||
The value for the :setting:`STATS_CLASS` setting is used by default.
|
||||
:type stats: :class:`scrapy.statscollectors.StatsCollector` instance or similar:
|
||||
any class that implements the `StatsCollector` interface
|
||||
If the value of the ``scheduler/unserializable`` stat is non-zero, consider
|
||||
enabling :setting:`SCHEDULER_DEBUG` to log a warning messages with details
|
||||
about the first unserializable request, to try and figure out how to make
|
||||
it serializable.
|
||||
|
||||
:param pqclass: A class to be used as priority queue for requests.
|
||||
The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default.
|
||||
:type pqclass: class
|
||||
|
||||
:param crawler: The crawler object corresponding to the current crawl.
|
||||
:type crawler: :class:`scrapy.crawler.Crawler`
|
||||
.. seealso:: :ref:`topics-jobs`
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
|
|
@ -198,9 +173,6 @@ class Scheduler(BaseScheduler):
|
|||
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler: Crawler) -> Self:
|
||||
"""
|
||||
Factory method, initializes the scheduler with arguments taken from the crawl settings
|
||||
"""
|
||||
dupefilter_cls = load_object(crawler.settings["DUPEFILTER_CLASS"])
|
||||
return cls(
|
||||
dupefilter=build_from_crawler(dupefilter_cls, crawler),
|
||||
|
|
@ -217,21 +189,12 @@ class Scheduler(BaseScheduler):
|
|||
return len(self) > 0
|
||||
|
||||
def open(self, spider: Spider) -> Deferred[None] | None:
|
||||
"""
|
||||
(1) initialize the memory queue
|
||||
(2) initialize the disk queue if the ``jobdir`` attribute is a valid directory
|
||||
(3) return the result of the dupefilter's ``open`` method
|
||||
"""
|
||||
self.spider: Spider = spider
|
||||
self.mqs: ScrapyPriorityQueue = self._mq()
|
||||
self.dqs: ScrapyPriorityQueue | None = self._dq() if self.dqdir else None
|
||||
return self.df.open()
|
||||
|
||||
def close(self, reason: str) -> Deferred[None] | None:
|
||||
"""
|
||||
(1) dump pending requests to disk if there is a disk queue
|
||||
(2) return the result of the dupefilter's ``close`` method
|
||||
"""
|
||||
if self.dqs is not None:
|
||||
state = self.dqs.close()
|
||||
assert isinstance(self.dqdir, str)
|
||||
|
|
@ -239,15 +202,6 @@ class Scheduler(BaseScheduler):
|
|||
return self.df.close(reason)
|
||||
|
||||
def enqueue_request(self, request: Request) -> bool:
|
||||
"""
|
||||
Unless the received request is filtered out by the Dupefilter, attempt to push
|
||||
it into the disk queue, falling back to pushing it into the memory queue.
|
||||
|
||||
Increment the appropriate stats, such as: ``scheduler/enqueued``,
|
||||
``scheduler/enqueued/disk``, ``scheduler/enqueued/memory``.
|
||||
|
||||
Return ``True`` if the request was stored successfully, ``False`` otherwise.
|
||||
"""
|
||||
if not request.dont_filter and self.df.request_seen(request):
|
||||
self.df.log(request, self.spider)
|
||||
return False
|
||||
|
|
@ -262,14 +216,6 @@ class Scheduler(BaseScheduler):
|
|||
return True
|
||||
|
||||
def next_request(self) -> Request | None:
|
||||
"""
|
||||
Return a :class:`~scrapy.Request` object from the memory queue,
|
||||
falling back to the disk queue if the memory queue is empty.
|
||||
Return ``None`` if there are no more enqueued requests.
|
||||
|
||||
Increment the appropriate stats, such as: ``scheduler/dequeued``,
|
||||
``scheduler/dequeued/disk``, ``scheduler/dequeued/memory``.
|
||||
"""
|
||||
request: Request | None = self.mqs.pop()
|
||||
assert self.stats is not None
|
||||
if request is not None:
|
||||
|
|
@ -283,9 +229,6 @@ class Scheduler(BaseScheduler):
|
|||
return request
|
||||
|
||||
def __len__(self) -> int:
|
||||
"""
|
||||
Return the total amount of enqueued requests
|
||||
"""
|
||||
return len(self.dqs) + len(self.mqs) if self.dqs is not None else len(self.mqs)
|
||||
|
||||
def _dqpush(self, request: Request) -> bool:
|
||||
|
|
|
|||
|
|
@ -130,6 +130,14 @@ class Request(object_ref):
|
|||
self._set_body(body)
|
||||
if not isinstance(priority, int):
|
||||
raise TypeError(f"Request priority not an integer: {priority!r}")
|
||||
|
||||
#: Value that the :ref:`scheduler <topics-scheduler>` may use for
|
||||
#: request prioritization.
|
||||
#:
|
||||
#: Built-in schedulers prioritize requests with a higher priority
|
||||
#: value.
|
||||
#:
|
||||
#: Negative values are allowed.
|
||||
self.priority: int = priority
|
||||
|
||||
if not (callable(callback) or callback is None):
|
||||
|
|
|
|||
|
|
@ -50,9 +50,9 @@ class QueueProtocol(Protocol):
|
|||
|
||||
|
||||
class ScrapyPriorityQueue:
|
||||
"""A priority queue implemented using multiple internal queues (typically,
|
||||
FIFO queues). It uses one internal queue for each priority value. The internal
|
||||
queue must implement the following methods:
|
||||
"""Default scheduler priority queue (:setting:`SCHEDULER_PRIORITY_QUEUE`).
|
||||
|
||||
The internal queue must implement the following methods:
|
||||
|
||||
* push(obj)
|
||||
* pop()
|
||||
|
|
@ -185,6 +185,13 @@ class DownloaderAwarePriorityQueue:
|
|||
"""PriorityQueue which takes Downloader activity into account:
|
||||
domains (slots) with the least amount of active downloads are dequeued
|
||||
first.
|
||||
|
||||
Another available type is
|
||||
``scrapy.pqueues.DownloaderAwarePriorityQueue``.
|
||||
``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than
|
||||
``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different
|
||||
domains in parallel. But currently ``scrapy.pqueues.DownloaderAwarePriorityQueue``
|
||||
does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
|
|
|
|||
Loading…
Reference in New Issue