From e75cd113d9d4c7afe1fe51bd73d71ad0f9f3a9b7 Mon Sep 17 00:00:00 2001 From: Pikmin1211 Date: Tue, 19 Oct 2021 16:27:26 -0400 Subject: [PATCH] generalization of ulimit for broad crawls --- docs/topics/broad-crawls.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index af69cc05c..19a5e41b5 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -82,11 +82,10 @@ should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. -.. note:: While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, - you may encounter ``OSError``s as the number of open files can exceed the default - Linux user limit (``ulimit``), which is typically ``1024``. In this case, you can - manually increase the Soft value of ``ulimit`` by using ``ulimit -S -n``. - +While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, +you may encounter OS specific errors regarding the number of currently open files. +Some systems allow this limit to be increased or removed. If you cannot do so, +you will need to reduce the :setting:`CONCURRENT_REQUESTS` value. Increase Twisted IO thread pool maximum size ============================================