RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge e75cd113d9 into c9446931a8

This commit is contained in:
JosephFarrell 2026-07-11 16:55:44 -07:00 committed by GitHub
parent c9446931a8 e75cd113d9
commit 97e1c71ff0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/topics/broad-crawls.rst
View File

@ -68,6 +68,10 @@ should pick a concurrency where CPU usage is at 80-90%.
Increasing concurrency also increases memory usage. If memory usage is a
concern, you might need to lower your global concurrency limit accordingly.
While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value,
you may encounter OS specific errors regarding the number of currently open files.
Some systems allow this limit to be increased or removed. If you cannot do so,
you will need to reduce the :setting:`CONCURRENT_REQUESTS` value.
Increase Twisted IO thread pool maximum size
============================================