diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index cace1f883..d6b9fd6f9 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -182,8 +182,9 @@ Be mindful of memory leaks ========================== If your broad crawl shows a high memory usage, in addition to :ref:`crawling in -BFO order ` and :ref:`lowering concurrency -` you should :ref:`debug your memory leaks +BFO order `, :ref:`lowering concurrency +` and :ref:`delaying start request iteration +` you should :ref:`debug your memory leaks `. diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index ceea2f7c0..f060710a4 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -158,6 +158,15 @@ scheduler_empty See :ref:`start-requests-lazy` for an example. + .. warning:: Only wait for this signal from + :meth:`~scrapy.Spider.start`. While no request can be sent, e.g. while + the responses being parsed exceed + :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`, the engine does not ask the + scheduler for requests, and hence this signal is not sent. So waiting + for it from a :ref:`callback ` can hang the crawl, + because the response being parsed is itself one of the responses that + may be blocking requests. + This signal does not support :ref:`asynchronous handlers `. diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index f2cfeb712..e68c208b7 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -396,8 +396,12 @@ Start requests Delaying start request iteration -------------------------------- -You can override the :meth:`~scrapy.Spider.start` method as follows to pause -its iteration whenever there are scheduled requests: +Scrapy iterates :meth:`~scrapy.Spider.start` as fast as it yields, so all start +requests reach the scheduler early in the crawl, however many they are. To +minimize the number of requests in the scheduler at any given time, and with it +resource usage (memory, or disk when using :setting:`JOBDIR`), override +:meth:`~scrapy.Spider.start` to pause its iteration whenever there are +scheduled requests: .. code-block:: python @@ -407,10 +411,6 @@ its iteration whenever there are scheduled requests: await self.crawler.signals.wait_for(signals.scheduler_empty) yield item_or_request -This can help minimize the number of requests in the scheduler at any given -time, to minimize resource usage (memory or disk, depending on -:setting:`JOBDIR`). - .. _builtin-spiders: Generic Spiders