From 0b2d220197fb7d9b274738cd812cb7d953f85c49 Mon Sep 17 00:00:00 2001 From: Adrian Date: Sun, 9 Aug 2026 11:19:50 +0200 Subject: [PATCH] Improve docs for multi-spider runs (#7907) --- docs/topics/practices.rst | 12 ++++++++++++ docs/topics/settings.rst | 5 +++++ 2 files changed, 17 insertions(+) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index dfa1e21f6..aeaf322c2 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -458,6 +458,18 @@ finishes before starting the next one: should not have a different value per spider, and :ref:`pre-crawler settings ` cannot be defined per spider. +Every other setting applies to each crawler separately. This includes +concurrency and politeness settings, such as :setting:`CONCURRENT_REQUESTS`, +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`, and +:ref:`AutoThrottle ` also throttles each crawler +separately. When crawling simultaneously, divide those values by the number of +crawlers to keep the combined load on your hardware and on target websites +unchanged. + +Because of this, running the same spider several times in the same process +multiplies those limits instead of increasing crawling capacity. To crawl +faster, raise :setting:`CONCURRENT_REQUESTS` on a single crawler. + .. seealso:: :ref:`run-from-script`. .. skip: end diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 793d65f35..dc09fdeeb 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -754,6 +754,11 @@ Default: ``60`` Timeout for processing of DNS queries in seconds. Float is supported. +The timeout starts when the query is queued into the Twisted reactor thread +pool, not when it is sent. If that thread pool is saturated, queries can time +out before being sent, in which case increasing +:setting:`REACTOR_THREADPOOL_MAXSIZE` helps more than increasing this setting. + .. note:: This setting is only used by :class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when