RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Mention scheduled requests as one place not to put responses in

This commit is contained in:
Adrián Chaves 2024-07-09 13:09:58 +02:00
parent ec7dbfd6a5
commit 3c5acd252f
1 changed files with 8 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
scrapy/core/downloader/__init__.py
View File

@ -197,12 +197,14 @@ class Downloader:
f"your memory allows it, you may increase "
f"RESPONSE_MAX_ACTIVE_SIZE, which should increase your "
f"crawl speed. If your code keeps non-weak references to "
f"Response objects, your crawl might get stuck "
f"indefinitely; you can set RESPONSE_MAX_ACTIVE_SIZE to 0 "
f"to disable this limit, but then your code might run out "
f"of memory. This message will only appear the first time "
f"this happens. To learn how often request processing has "
f"been paused during a crawl for this reason, see the "
f"Response objects, e.g. in (scheduled) requests or in a "
f"container within a component, your crawl might get "
f"stuck indefinitely; you can set "
f"RESPONSE_MAX_ACTIVE_SIZE to 0 to disable this limit, "
f"but then your code might run out of memory. This "
f"message will only appear the first time this happens. "
f"To learn how often request processing has been paused "
f"during a crawl for this reason, see the "
f"request_backouts/response_max_active_size stat."
)
self._count_backout("response_max_active_size")