From 852ad0b3a893d6e3be2547f039a48347a76ee150 Mon Sep 17 00:00:00 2001
From: Pikmin1211 <92343322+JosephFarrell@users.noreply.github.com>
Date: Mon, 11 Oct 2021 20:48:56 -0400
Subject: [PATCH] ulimit for broad crawls

issue #5259
Information / Documentation on Linux's ulimit:
https://www.ibm.com/docs/cs/aix/7.1?topic=u-ulimit-command
https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user
---
 docs/topics/broad-crawls.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst
index 63b60312e..af69cc05c 100644
--- a/docs/topics/broad-crawls.rst
+++ b/docs/topics/broad-crawls.rst
@@ -82,6 +82,11 @@ should pick a concurrency where CPU usage is at 80-90%.
 Increasing concurrency also increases memory usage. If memory usage is a
 concern, you might need to lower your global concurrency limit accordingly.
 
+.. note:: While performing a broad crawl with a high :setting:`CONCURRENT_REQUESTS` value,
+          you may encounter ``OSError``s as the number of open files can exceed the default
+          Linux user limit (``ulimit``), which is typically ``1024``. In this case, you can 
+          manually increase the Soft value of ``ulimit`` by using ``ulimit -S -n``.
+
 
 Increase Twisted IO thread pool maximum size
 ============================================