RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ulimit for broad crawls 

issue #5259
Information / Documentation on Linux's ulimit:
https://www.ibm.com/docs/cs/aix/7.1?topic=u-ulimit-command
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user
This commit is contained in:
Pikmin1211 2021-10-11 20:48:56 -04:00
parent 65d60b9692
commit 852ad0b3a8
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/topics/broad-crawls.rst
View File

@ -82,6 +82,11 @@ should pick a concurrency where CPU usage is at 80-90%.
Increasing concurrency also increases memory usage. If memory usage is a
concern, you might need to lower your global concurrency limit accordingly.
.. note:: While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value,
you may encounter ``OSError``s as the number of open files can exceed the default
Linux user limit (``ulimit``), which is typically ``1024``. In this case, you can
manually increase the Soft value of ``ulimit`` by using ``ulimit -S -n``.
Increase Twisted IO thread pool maximum size
============================================