2.9 KiB
AutoThrottle extension
This is an extension for automatically throttling crawling speed based on load of both the Scrapy server and the website you are crawling.
Design goals
- be nicer to sites instead of using default download delay of zero
- automatically adjust scrapy to the optimum crawling speed, so the user doesn't have to tune the download delays and concurrent requests to find the optimum one. The user only needs to specify the maximum concurrent requests it allows, and the extension does the rest.
How it works
In Scrapy, the download latency is measured as the time elapsed between establishing the TCP connection and receiving the HTTP headers.
Note that these latencies are very hard to measure accurately in a cooperative multitasking environment because Scrapy may be busy processing a spider callback, for example, and unable to attend downloads. However, these latencies should still give a reasonable estimate of how busy Scrapy (and ultimately, the server) is, and this extension builds on that premise.
Throttling algorithm
This adjusts download delays and concurrency based on the following rules:
spiders always start with one concurrent request and a download delay of :setting:`AUTOTHROTTLE_START_DELAY`
System Message: ERROR/3 (<stdin>, line 38); backlink
Unknown interpreted text role "setting".
when a response is received, the download delay is adjusted to the average of previous download delay and the latency of the response.
Note
The AutoThrottle extension honours the standard Scrapy settings for concurrency and delay. This means that it will never set a download delay lower than :setting:`DOWNLOAD_DELAY` or a concurrency higher than :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (or :setting:`CONCURRENT_REQUESTS_PER_IP`, depending on which one you use).
System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 43); backlink
Unknown interpreted text role "setting".Settings
The settings used to control the AutoThrottle extension are:
:setting:`AUTOTHROTTLE_ENABLED`
System Message: ERROR/3 (<stdin>, line 54); backlink
Unknown interpreted text role "setting".
:setting:`AUTOTHROTTLE_START_DELAY`
System Message: ERROR/3 (<stdin>, line 55); backlink
Unknown interpreted text role "setting".
:setting:`AUTOTHROTTLE_MAX_DELAY`
System Message: ERROR/3 (<stdin>, line 56); backlink
Unknown interpreted text role "setting".
-
System Message: ERROR/3 (<stdin>, line 57); backlink
Unknown interpreted text role "setting".
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`
System Message: ERROR/3 (<stdin>, line 58); backlink
Unknown interpreted text role "setting".
:setting:`CONCURRENT_REQUESTS_PER_IP`
System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "setting".
-
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "setting".
For more information see :ref:`autothrottle-algorithm`.
System Message: ERROR/3 (<stdin>, line 62); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 64)
Unknown directive type "setting".
.. setting:: AUTOTHROTTLE_ENABLED
AUTOTHROTTLE_ENABLED
Default: False
Enables the AutoThrottle extension.
System Message: ERROR/3 (<stdin>, line 73)
Unknown directive type "setting".
.. setting:: AUTOTHROTTLE_START_DELAY
AUTOTHROTTLE_START_DELAY
Default: 5.0
The initial download delay (in seconds).
System Message: ERROR/3 (<stdin>, line 82)
Unknown directive type "setting".
.. setting:: AUTOTHROTTLE_MAX_DELAY
AUTOTHROTTLE_MAX_DELAY
Default: 60.0
The maximum download delay (in seconds) to be set in case of high latencies.
System Message: ERROR/3 (<stdin>, line 91)
Unknown directive type "setting".
.. setting:: AUTOTHROTTLE_DEBUG
AUTOTHROTTLE_DEBUG
Default: False
Enable AutoThrottle debug mode which will display stats on every response received, so you can see how the throttling parameters are being adjusted in real time.