RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improve docs for multi-spider runs (#7907)

This commit is contained in:
Adrian 2026-08-09 11:19:50 +02:00 committed by GitHub
parent 0e324f3d4a
commit 0b2d220197
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 17 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
docs/topics/practices.rst
View File

@ -458,6 +458,18 @@ finishes before starting the next one:
should not have a different value per spider, and :ref:`pre-crawler
settings <pre-crawler-settings>` cannot be defined per spider.
Every other setting applies to each crawler separately. This includes
concurrency and politeness settings, such as :setting:`CONCURRENT_REQUESTS`,
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`, and
:ref:`AutoThrottle <topics-autothrottle>` also throttles each crawler
separately. When crawling simultaneously, divide those values by the number of
crawlers to keep the combined load on your hardware and on target websites
unchanged.
Because of this, running the same spider several times in the same process
multiplies those limits instead of increasing crawling capacity. To crawl
faster, raise :setting:`CONCURRENT_REQUESTS` on a single crawler.
.. seealso:: :ref:`run-from-script`.
.. skip: end

5
docs/topics/settings.rst
View File

@ -754,6 +754,11 @@ Default: ``60``
Timeout for processing of DNS queries in seconds. Float is supported.
The timeout starts when the query is queued into the Twisted reactor thread
pool, not when it is sent. If that thread pool is saturated, queries can time
out before being sent, in which case increasing
:setting:`REACTOR_THREADPOOL_MAXSIZE` helps more than increasing this setting.
.. note::
This setting is only used by
:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when