diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst index 39bd13747..62bf2c835 100644 --- a/docs/topics/optimize.rst +++ b/docs/topics/optimize.rst @@ -21,11 +21,12 @@ For broad crawls, consider these adjustments: - .. _broad-crawls-concurrency: - Increase global concurrency: + Increase the global concurrency: - - Set :setting:`CONCURRENT_REQUESTS` as close to “number of target - domains” times :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (e.g. 80 for - 10 domains) as your CPU and memory allow. + - Set :setting:`CONCURRENT_REQUESTS` as close to + :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains] + (e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory + allow. - Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing :setting:`CONCURRENT_REQUESTS` stops making a difference. @@ -87,8 +88,9 @@ Lowering memory usage :attr:`~scrapy.Request.callback` cannot yield additional requests. - - If you have many :ref:`start requests `, use :ref:`lazy - ` or :ref:`idle ` scheduling. + - If you have multiple :ref:`start requests `, consider + :ref:`lazy ` or :ref:`idle ` + scheduling. - Set :setting:`JOBDIR` to offload all scheduled requests to disk. diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a98f77df5..d00b11919 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -20,6 +20,7 @@ functionality not required in the base classes. These are described below in :ref:`topics-request-response-ref-request-subclasses` and :ref:`topics-request-response-ref-response-subclasses`. +.. _request: Request objects =============== diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 0c05b34ea..1822ecbb6 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -369,13 +369,16 @@ See `Scrapyd documentation`_. Start requests ============== -**Start requests** are the requests yielded by the :meth:`~scrapy.Spider.start` -method of a spider. +**Start requests** are the :ref:`requests ` that serve as a starting +point for a spider. -By default, Scrapy does not try to send :meth:`~scrapy.Spider.start` requests -in order. Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and -:ref:`scheduling ` start requests. However, you can change -that. +They are yielded by the :meth:`~scrapy.Spider.start` method, and may be +modified by :ref:`spider middlewares `. + +They are not necessarily the *first* requests sent, and may not be sent in +order; reaching :setting:`CONCURRENT_REQUESTS` and :ref:`scheduling +` start requests is prioritized by default. But you can +change that. .. The request send order when all start requests and callback requests have @@ -454,8 +457,8 @@ is reached, start requests are removed from the scheduler immediately after they are scheduled. If you do not yield the highest-priority start requests first, and you want -request priorities to be respected since the first request, pause the scheduler -while yielding your start requests: +request priorities to be respected since the very first request, pause the +:ref:`scheduler ` while yielding your start requests: .. code-block:: python