Minor edits around Optimization

This commit is contained in:
Adrián Chaves 2025-03-31 17:13:40 +02:00
parent a858ddfef9
commit 28df50ddcc
3 changed files with 20 additions and 14 deletions

View File

@ -21,11 +21,12 @@ For broad crawls, consider these adjustments:
- .. _broad-crawls-concurrency:
Increase global concurrency:
Increase the global concurrency:
- Set :setting:`CONCURRENT_REQUESTS` as close to “number of target
domains” times :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (e.g. 80 for
10 domains) as your CPU and memory allow.
- Set :setting:`CONCURRENT_REQUESTS` as close to
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains]
(e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory
allow.
- Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing
:setting:`CONCURRENT_REQUESTS` stops making a difference.
@ -87,8 +88,9 @@ Lowering memory usage
:attr:`~scrapy.Request.callback` cannot yield additional
requests.
- If you have many :ref:`start requests <start-requests>`, use :ref:`lazy
<start-requests-lazy>` or :ref:`idle <start-requests-idle>` scheduling.
- If you have multiple :ref:`start requests <start-requests>`, consider
:ref:`lazy <start-requests-lazy>` or :ref:`idle <start-requests-idle>`
scheduling.
- Set :setting:`JOBDIR` to offload all scheduled requests to disk.

View File

@ -20,6 +20,7 @@ functionality not required in the base classes. These are described
below in :ref:`topics-request-response-ref-request-subclasses` and
:ref:`topics-request-response-ref-response-subclasses`.
.. _request:
Request objects
===============

View File

@ -369,13 +369,16 @@ See `Scrapyd documentation`_.
Start requests
==============
**Start requests** are the requests yielded by the :meth:`~scrapy.Spider.start`
method of a spider.
**Start requests** are the :ref:`requests <request>` that serve as a starting
point for a spider.
By default, Scrapy does not try to send :meth:`~scrapy.Spider.start` requests
in order. Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and
:ref:`scheduling <topics-scheduler>` start requests. However, you can change
that.
They are yielded by the :meth:`~scrapy.Spider.start` method, and may be
modified by :ref:`spider middlewares <topics-spider-middleware>`.
They are not necessarily the *first* requests sent, and may not be sent in
order; reaching :setting:`CONCURRENT_REQUESTS` and :ref:`scheduling
<topics-scheduler>` start requests is prioritized by default. But you can
change that.
..
The request send order when all start requests and callback requests have
@ -454,8 +457,8 @@ is reached, start requests are removed from the scheduler immediately after
they are scheduled.
If you do not yield the highest-priority start requests first, and you want
request priorities to be respected since the first request, pause the scheduler
while yielding your start requests:
request priorities to be respected since the very first request, pause the
:ref:`scheduler <topics-scheduler>` while yielding your start requests:
.. code-block:: python