From 34e4ed72ea87601ce17c8d248d6a54d3e8b73194 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 21 Jun 2022 12:46:54 +0200 Subject: [PATCH] Document how DOWNLOAD_DELAY affects per-domain concurrency --- docs/topics/settings.rst | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 2046c6446..468975d6a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -633,25 +633,41 @@ DOWNLOAD_DELAY Default: ``0`` -The amount of time (in secs) that the downloader should wait before downloading -consecutive pages from the same website. This can be used to throttle the -crawling speed to avoid hitting servers too hard. Decimal numbers are -supported. Example:: +Minimum seconds to wait between 2 consecutive requests to the same domain. - DOWNLOAD_DELAY = 0.25 # 250 ms of delay +Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting +servers too hard. + +Decimal numbers are supported. For example, to send a maximum of 4 requests +every 10 seconds:: + + DOWNLOAD_DELAY = 2.5 This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY` -setting (which is enabled by default). By default, Scrapy doesn't wait a fixed -amount of time between requests, but uses a random interval between 0.5 * :setting:`DOWNLOAD_DELAY` and 1.5 * :setting:`DOWNLOAD_DELAY`. +setting, which is enabled by default. When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced -per ip address instead of per domain. +per IP address instead of per domain. + +Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain +concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response +time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective +concurrency for that domain is 1. When testing throttling configurations, it +usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first, +and only increase :setting:`DOWNLOAD_DELAY` once +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is +desired. .. _spider-download_delay-attribute: -You can also change this setting per spider by setting ``download_delay`` +You can change this setting per spider by setting the ``download_delay`` spider attribute. +It is also possible to change this setting per domain, although it requires +non-trivial code. See the implementation of the :ref:`AutoThrottle +` extension for an example. + + .. setting:: DOWNLOAD_HANDLERS DOWNLOAD_HANDLERS