RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/autothrottle.rst

6.9 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

AutoThrottle extension

This is an extension for automatically throttling crawling speed based on load of both the Scrapy server and the website you are crawling.

Design goals

  1. be nicer to sites instead of using default download delay of zero
  2. automatically adjust Scrapy to the optimum crawling speed, so the user doesn't have to tune the download delays to find the optimum one. The user only needs to specify the maximum concurrent requests it allows, and the extension does the rest.

How it works

Scrapy allows defining the concurrency and delay of different download slots, e.g. through the :setting:`DOWNLOAD_SLOTS` setting. By default requests are assigned to slots based on their URL domain, although it is possible to customize the download slot of any request.

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "setting".

The AutoThrottle extension adjusts the delay of each download slot dynamically, to make your spider send :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent requests on average to each remote website.

System Message: ERROR/3 (<stdin>, line 29); backlink

Unknown interpreted text role "setting".

It uses download latency to compute the delays. The main idea is the following: if a server needs latency seconds to respond, a client should send a request each latency/N seconds to have N requests processed in parallel.

Instead of adjusting the delays one can just set a small fixed download delay and impose hard limits on concurrency using :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. It will provide a similar effect, but there are some important differences:

System Message: ERROR/3 (<stdin>, line 38); backlink

Unknown interpreted text role "setting".
  • because the download delay is small there will be occasional bursts of requests;
  • often non-200 (error) responses can be returned faster than regular responses, so with a small download delay and a hard concurrency limit crawler will be sending requests to server faster when server starts to return errors. But this is an opposite of what crawler should do - in case of errors it makes more sense to slow down: these errors may be caused by the high request rate.

AutoThrottle doesn't have these issues.

Throttling algorithm

AutoThrottle algorithm adjusts download delays based on the following rules:

  1. spiders always start with a download delay of :setting:`AUTOTHROTTLE_START_DELAY`;

    System Message: ERROR/3 (<stdin>, line 59); backlink

    Unknown interpreted text role "setting".

  2. when a response is received, the target download delay is calculated as latency / N where latency is a latency of the response, and N is :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY`.

    System Message: ERROR/3 (<stdin>, line 61); backlink

    Unknown interpreted text role "setting".

  3. download delay for next requests is set to the average of previous download delay and the target download delay;

  4. latencies of non-200 responses are not allowed to decrease the delay;

  5. download delay can't become less than :setting:`DOWNLOAD_DELAY` or greater than :setting:`AUTOTHROTTLE_MAX_DELAY`

    System Message: ERROR/3 (<stdin>, line 67); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 67); backlink

    Unknown interpreted text role "setting".

Note

The AutoThrottle extension honours the standard Scrapy settings for concurrency and delay. This means that it will respect :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and never set a download delay lower than :setting:`DOWNLOAD_DELAY`.

System Message: ERROR/3 (<stdin>, line 70); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 70); backlink

Unknown interpreted text role "setting".

In Scrapy, the download latency is measured as the time elapsed between sending the request and receiving the HTTP headers.

Note that these latencies are very hard to measure accurately in a cooperative multitasking environment because Scrapy may be busy processing a spider callback, for example, and unable to attend downloads. However, these latencies should still give a reasonable estimate of how busy Scrapy (and ultimately, the server) is, and this extension builds on that premise.

System Message: ERROR/3 (<stdin>, line 86)

Unknown directive type "reqmeta".

.. reqmeta:: autothrottle_dont_adjust_delay

Prevent specific requests from triggering slot delay adjustments

System Message: ERROR/3 (<stdin>, line 91)

Unknown directive type "versionadded".

.. versionadded:: 2.12.0

AutoThrottle adjusts the delay of download slots based on the latencies of responses that belong to that download slot. The only exceptions are non-200 responses, which are only taken into account to increase that delay, but ignored if they would decrease that delay.

You can also set the autothrottle_dont_adjust_delay request metadata key to True in any request to prevent its response latency from impacting the delay of its download slot:

System Message: WARNING/2 (<stdin>, line 102)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy import Request

    Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})

Note, however, that AutoThrottle still determines the starting delay of every download slot by setting the download_delay attribute on the running spider. If you want AutoThrottle not to impact a download slot at all, in addition to setting this meta key in all requests that use that download slot, you might want to set a custom value for the delay attribute of that download slot, e.g. using :setting:`DOWNLOAD_SLOTS`.

System Message: ERROR/3 (<stdin>, line 108); backlink

Unknown interpreted text role "setting".

Settings

The settings used to control the AutoThrottle extension are:

For more information see :ref:`autothrottle-algorithm`.

System Message: ERROR/3 (<stdin>, line 128); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 130)

Unknown directive type "setting".

.. setting:: AUTOTHROTTLE_ENABLED

AUTOTHROTTLE_ENABLED

Default: False

Enables the AutoThrottle extension.

System Message: ERROR/3 (<stdin>, line 139)

Unknown directive type "setting".

.. setting:: AUTOTHROTTLE_START_DELAY

AUTOTHROTTLE_START_DELAY

Default: 5.0

The initial download delay (in seconds).

System Message: ERROR/3 (<stdin>, line 148)

Unknown directive type "setting".

.. setting:: AUTOTHROTTLE_MAX_DELAY

AUTOTHROTTLE_MAX_DELAY

Default: 60.0

The maximum download delay (in seconds) to be set in case of high latencies.

System Message: ERROR/3 (<stdin>, line 157)

Unknown directive type "setting".

.. setting:: AUTOTHROTTLE_TARGET_CONCURRENCY

AUTOTHROTTLE_TARGET_CONCURRENCY

Default: 1.0

Average number of requests Scrapy should be sending in parallel to remote websites. It must be higher than 0.0.

By default, AutoThrottle adjusts the delay to send a single concurrent request to each of the remote websites. Set this option to a higher value (e.g. 2.0) to increase the throughput and the load on remote servers. A lower AUTOTHROTTLE_TARGET_CONCURRENCY value (e.g. 0.5) makes the crawler more conservative and polite.

Note that :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is still respected when AutoThrottle extension is enabled. This means that if AUTOTHROTTLE_TARGET_CONCURRENCY is set to a value higher than :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, the crawler won't reach this number of concurrent requests.

System Message: ERROR/3 (<stdin>, line 173); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 173); backlink

Unknown interpreted text role "setting".

At every given time point Scrapy can be sending more or less concurrent requests than AUTOTHROTTLE_TARGET_CONCURRENCY; it is a suggested value the crawler tries to approach, not a hard limit.

System Message: ERROR/3 (<stdin>, line 183)

Unknown directive type "setting".

.. setting:: AUTOTHROTTLE_DEBUG

AUTOTHROTTLE_DEBUG

Default: False

Enable AutoThrottle debug mode which will display stats on every response received, so you can see how the throttling parameters are being adjusted in real time.

</html>