From 852ad0b3a893d6e3be2547f039a48347a76ee150 Mon Sep 17 00:00:00 2001 From: Pikmin1211 <92343322+JosephFarrell@users.noreply.github.com> Date: Mon, 11 Oct 2021 20:48:56 -0400 Subject: [PATCH 1/2] ulimit for broad crawls issue #5259 Information / Documentation on Linux's ulimit: https://www.ibm.com/docs/cs/aix/7.1?topic=u-ulimit-command https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user --- docs/topics/broad-crawls.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 63b60312e..af69cc05c 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -82,6 +82,11 @@ should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. +.. note:: While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, + you may encounter ``OSError``s as the number of open files can exceed the default + Linux user limit (``ulimit``), which is typically ``1024``. In this case, you can + manually increase the Soft value of ``ulimit`` by using ``ulimit -S -n``. + Increase Twisted IO thread pool maximum size ============================================ From e75cd113d9d4c7afe1fe51bd73d71ad0f9f3a9b7 Mon Sep 17 00:00:00 2001 From: Pikmin1211 Date: Tue, 19 Oct 2021 16:27:26 -0400 Subject: [PATCH 2/2] generalization of ulimit for broad crawls --- docs/topics/broad-crawls.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index af69cc05c..19a5e41b5 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -82,11 +82,10 @@ should pick a concurrency where CPU usage is at 80-90%. Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly. -.. note:: While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, - you may encounter ``OSError``s as the number of open files can exceed the default - Linux user limit (``ulimit``), which is typically ``1024``. In this case, you can - manually increase the Soft value of ``ulimit`` by using ``ulimit -S -n``. - +While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value, +you may encounter OS specific errors regarding the number of currently open files. +Some systems allow this limit to be increased or removed. If you cannot do so, +you will need to reduce the :setting:`CONCURRENT_REQUESTS` value. Increase Twisted IO thread pool maximum size ============================================