diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index cace1f883..e99b0932a 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -68,6 +68,10 @@ should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. +While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, +you may encounter OS specific errors regarding the number of currently open files. +Some systems allow this limit to be increased or removed. If you cannot do so, +you will need to reduce the :setting:`CONCURRENT_REQUESTS` value. Increase Twisted IO thread pool maximum size ============================================