RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improve the docs about delaying start request iteration (#7883)

This commit is contained in:
Adrian 2026-08-09 13:57:08 +02:00 committed by GitHub
parent 59ce27afdd
commit 050a8cf159
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 18 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/topics/broad-crawls.rst
View File

@ -182,8 +182,9 @@ Be mindful of memory leaks
==========================
If your broad crawl shows a high memory usage, in addition to :ref:`crawling in
BFO order <broad-crawls-bfo>` and :ref:`lowering concurrency
<broad-crawls-concurrency>` you should :ref:`debug your memory leaks
BFO order <broad-crawls-bfo>`, :ref:`lowering concurrency
<broad-crawls-concurrency>` and :ref:`delaying start request iteration
<start-requests-lazy>` you should :ref:`debug your memory leaks
<topics-leaks>`.

9
docs/topics/signals.rst
View File

@ -158,6 +158,15 @@ scheduler_empty
See :ref:`start-requests-lazy` for an example.
.. warning:: Only wait for this signal from
:meth:`~scrapy.Spider.start`. While no request can be sent, e.g. while
the responses being parsed exceed
:setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`, the engine does not ask the
scheduler for requests, and hence this signal is not sent. So waiting
for it from a :ref:`callback <callbacks>` can hang the crawl,
because the response being parsed is itself one of the responses that
may be blocking requests.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.

12
docs/topics/spiders.rst
View File

@ -396,8 +396,12 @@ Start requests
Delaying start request iteration
--------------------------------
You can override the :meth:`~scrapy.Spider.start` method as follows to pause
its iteration whenever there are scheduled requests:
Scrapy iterates :meth:`~scrapy.Spider.start` as fast as it yields, so all start
requests reach the scheduler early in the crawl, however many they are. To
minimize the number of requests in the scheduler at any given time, and with it
resource usage (memory, or disk when using :setting:`JOBDIR`), override
:meth:`~scrapy.Spider.start` to pause its iteration whenever there are
scheduled requests:
.. code-block:: python
@ -407,10 +411,6 @@ its iteration whenever there are scheduled requests:
await self.crawler.signals.wait_for(signals.scheduler_empty)
yield item_or_request
This can help minimize the number of requests in the scheduler at any given
time, to minimize resource usage (memory or disk, depending on
:setting:`JOBDIR`).
.. _builtin-spiders:
Generic Spiders