40 KiB

Raw Blame History

Throttling

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "versionadded".

.. versionadded:: VERSION

Sending too many requests too quickly can overwhelm websites. :ref:`Throttling <basic-throttling>` and :ref:`backoff <backoff>` aim to prevent that.

System Message: ERROR/3 (<stdin>, line 9); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 9); backlink

Unknown interpreted text role "ref".

Concurrency and delay

Requests are throttled on a per-domain basis by default [1]. This allows efficient crawling of multiple sites simultaneously.

Each domain and subdomain is treated separately: requests to books.toscrape.com and quotes.toscrape.com each have their own throttling limits, as do toscrape.com and books.toscrape.com.

The main throttling :ref:`settings <topics-settings>` are:

System Message: ERROR/3 (<stdin>, line 27); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 29)

Unknown directive type "setting".
```
.. setting:: CONCURRENT_REQUESTS_PER_DOMAIN
```
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (default: 8)

System Message: ERROR/3 (<stdin>, line 31); backlink

Unknown interpreted text role "setting".

Maximum number of simultaneous requests per domain.

It defines a number of “slots” per domain. Each slot can send 1 request at a time: it sends a request, waits for the response, then sends the next request, and so on.
System Message: ERROR/3 (<stdin>, line 39)

Unknown directive type "setting".
```
.. setting:: DOWNLOAD_DELAY
```
:setting:`DOWNLOAD_DELAY` (default: 1 (:ref:`fallback <default-settings>`: 0))

System Message: ERROR/3 (<stdin>, line 41); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 41); backlink

Unknown interpreted text role "ref".

Minimum seconds between any two requests to the same domain.

To target a specific number of requests per minute (RPM) per domain, set this to 60 / RPM. For example, DOWNLOAD_DELAY = 1.0 for 60 RPM, or DOWNLOAD_DELAY = 2.0 for 30 RPM.

When configuring these settings, note that:

:setting:`CONCURRENT_REQUESTS` caps :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`.

System Message: ERROR/3 (<stdin>, line 52); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 52); backlink

Unknown interpreted text role "setting".
If DOWNLOAD_DELAY ≥ response time, concurrency is effectively 1, because the next request to the domain is not sent until the delay elapses, by which time the previous response has already arrived.

[1]

(1, 2)

You can :ref:`customize <throttling-scopes>` how requests are grouped for throttling, but domain-based throttling works well in most cases. For more complex domain grouping strategies, see :ref:`alternative-domain-throttling`.

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 64)

Unknown directive type "setting".

.. setting:: THROTTLING_SCOPES

Per-domain throttling

The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling behavior for specific domains [1].

System Message: ERROR/3 (<stdin>, line 71); backlink

Unknown interpreted text role "setting".

It is a dict that maps scope names to :class:`~scrapy.throttling.ThrottlingScopeConfig` dicts.

System Message: ERROR/3 (<stdin>, line 74); backlink

Unknown interpreted text role "class".

Its default value allows faster crawling of the testing website using during the :ref:`tutorial <intro-tutorial>` while maintaining conservative defaults for other domains:

System Message: ERROR/3 (<stdin>, line 77); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 81)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    THROTTLING_SCOPES = {
        "quotes.toscrape.com": {"concurrency": 16, "delay": 0.0},
    }

Additional keys like "jitter" and "backoff" can be used here and are covered later on.

Backoff

When servers respond with rate limiting errors (like HTTP 429) or network timeouts occur, request rate is automatically reduced using exponential backoff.

The key settings are:

System Message: ERROR/3 (<stdin>, line 104)

Unknown directive type "setting".
```
.. setting:: BACKOFF_HTTP_CODES
```
:setting:`BACKOFF_HTTP_CODES` (default: [429, 502, 503, 504, 520, 521, 522, 523, 524])

System Message: ERROR/3 (<stdin>, line 106); backlink

Unknown interpreted text role "setting".

HTTP response status codes that trigger backoff.
System Message: ERROR/3 (<stdin>, line 110)

Unknown directive type "setting".
```
.. setting:: BACKOFF_DELAY_FACTOR
```
:setting:`BACKOFF_DELAY_FACTOR` (default: 2.0)

System Message: ERROR/3 (<stdin>, line 112); backlink

Unknown interpreted text role "setting".

Factor by which the delay of a scope is multiplied on each backoff step (2×, 4×, 8×, etc.).
System Message: ERROR/3 (<stdin>, line 117)

Unknown directive type "setting".
```
.. setting:: BACKOFF_MAX_DELAY
```
:setting:`BACKOFF_MAX_DELAY` (default: 300.0)

System Message: ERROR/3 (<stdin>, line 119); backlink

Unknown interpreted text role "setting".

Maximum delay, in seconds, that backoff can reach. Also caps :ref:`Retry-After <retry-after>` and :ref:`RateLimit-Reset <rate-limiting-headers>` delays.

System Message: ERROR/3 (<stdin>, line 121); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 121); backlink

Unknown interpreted text role "ref".

See :ref:`throttling-settings` for :setting:`BACKOFF_EXCEPTIONS`, :setting:`BACKOFF_JITTER`, :setting:`BACKOFF_MIN_DELAY` and :setting:`BACKOFF_WINDOW`.

System Message: ERROR/3 (<stdin>, line 125); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 125); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 125); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 125); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 125); backlink

Unknown interpreted text role "setting".

How backoff works

Every :ref:`throttling scope <throttling-scopes>` keeps a current delay that starts at its configured value (0 by default, or :setting:`DOWNLOAD_DELAY` for the default per-domain scopes).

System Message: ERROR/3 (<stdin>, line 134); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 134); backlink

Unknown interpreted text role "setting".

A backoff trigger is a response whose status code is in :setting:`BACKOFF_HTTP_CODES` or a download exception whose type is in :setting:`BACKOFF_EXCEPTIONS`. On each trigger:

System Message: ERROR/3 (<stdin>, line 138); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 138); backlink

Unknown interpreted text role "setting".

If the response carries a :ref:`Retry-After or RateLimit-Reset <rate-limiting-headers>` value, that value (capped at :setting:`BACKOFF_MAX_DELAY`) is honored as a hard minimum: no request is sent for the scope until it elapses.

System Message: ERROR/3 (<stdin>, line 142); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 142); backlink

Unknown interpreted text role "setting".
Otherwise the delay grows exponentially:
```
delay = min(BACKOFF_MAX_DELAY, max(BACKOFF_MIN_DELAY, delay * BACKOFF_DELAY_FACTOR))
```
:setting:`BACKOFF_JITTER` is then applied so that requests that backed off together do not retry in lockstep.

System Message: ERROR/3 (<stdin>, line 153); backlink

Unknown interpreted text role "setting".

Recovery is linear: after a scope goes a full :setting:`BACKOFF_WINDOW` without a new trigger, its delay drops by one :setting:`BACKOFF_DELAY_FACTOR` step toward the configured value, and keeps dropping one step per quiet window until it is back to the configured value. A new trigger resets the countdown.

System Message: ERROR/3 (<stdin>, line 156); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 156); backlink

Unknown interpreted text role "setting".

This exponential-increase / linear-decrease pattern, similar to TCP congestion control, makes a scope back off quickly when a server is unhappy and return to full speed gradually once it recovers. To keep a scope hovering around a target rate instead of repeatedly probing and backing off, enable :ref:`rampup <rampup>`.

System Message: ERROR/3 (<stdin>, line 161); backlink

Unknown interpreted text role "ref".

Per-scope backoff configuration

The global BACKOFF_* settings can be overridden per scope with the "backoff" key of a :setting:`THROTTLING_SCOPES` entry, an instance of :class:`~scrapy.throttling.BackoffConfig`:

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 176)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "example.com": {
            "backoff": {
                "http_codes": [429, 503],
                "exceptions": ["builtins.IOError"],
                "delay_factor": 1.2,
                "max_delay": 180.0,
                "min_delay": 5.0,
                "jitter": [0.01, 0.33],
            },
        },
    }

Any key left out falls back to the matching global BACKOFF_* setting.

Rampup

When using APIs that charge per request, like web scraping APIs, you often want to maximize throughput while staying within rate limits. To do that, set "rampup" to True in :setting:`THROTTLING_SCOPES`:

System Message: ERROR/3 (<stdin>, line 199); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 203)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "rampup": True,
        },
    }

Rampup increases concurrency or lowers delay as needed based on the following setting:

System Message: ERROR/3 (<stdin>, line 215)

Unknown directive type "setting".
```
.. setting:: RAMPUP_BACKOFF_TARGET
```
:setting:`RAMPUP_BACKOFF_TARGET` (default: 1)

System Message: ERROR/3 (<stdin>, line 217); backlink

Unknown interpreted text role "setting".

Target number of backoff responses per :setting:`BACKOFF_WINDOW` that :ref:`rampup <rampup>` aims for when probing the rate limit of a scope. Can be a range like [1, 3].

System Message: ERROR/3 (<stdin>, line 219); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 219); backlink

Unknown interpreted text role "ref".

For every :setting:`BACKOFF_WINDOW` that stays below :setting:`RAMPUP_BACKOFF_TARGET` backoff triggers, rampup increases throughput one step: it first lowers the delay, and once the delay reaches its minimum it raises the concurrency limit above the "min_concurrency" floor of the scope. Windows that hit the target hold the current rate, and windows that exceed it let normal :ref:`backoff <backoff>` reduce the rate. The result is a rate that converges on roughly :setting:`RAMPUP_BACKOFF_TARGET` rate-limit responses per window — the most throughput a scope allows without being penalized.

System Message: ERROR/3 (<stdin>, line 223); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 223); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 223); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 223); backlink

Unknown interpreted text role "setting".

Rampup behavior can be fine-tuned per scope by giving "rampup" a dict instead of True:

System Message: ERROR/3 (<stdin>, line 236)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "rampup": {
                "backoff_target": [1, 3],  # overrides RAMPUP_BACKOFF_TARGET
                "delay_factor": 0.5,  # multiply the delay by this on each ramp-up step
                "min_delay": 0.05,  # do not ramp the delay below this
            },
        },
    }

Rate limiting headers

Servers may include Retry-After or RateLimit-Reset headers to indicate when you should make your next request. These headers are respected automatically during :ref:`backoff <backoff>`, using their values as minimum delays (capped at :setting:`BACKOFF_MAX_DELAY`).

System Message: ERROR/3 (<stdin>, line 256); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 256); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 264)

Unknown directive type "seealso".

.. seealso:: :setting:`REDIRECT_MAX_DELAY`

robots.txt

Crawl-Delay is a non-standard robots.txt directive that indicates a number of seconds to wait between requests.

System Message: ERROR/3 (<stdin>, line 275)

Unknown directive type "setting".

.. setting:: THROTTLING_ROBOTSTXT_OBEY

System Message: ERROR/3 (<stdin>, line 276)

Unknown directive type "setting".

.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY

If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLING_ROBOTSTXT_OBEY` are True (default), valid Crawl-Delay directives override :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`. Concurrency is set to 1 and delay is set to the value of Crawl-Delay, capped at :setting:`THROTTLING_ROBOTSTXT_MAX_DELAY` (default: 60.0).

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "setting".

If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it will be respected, but a warning will be logged about the discrepancy with Crawl-Delay. Set ignore_robots_txt to True to silence this warning.

System Message: ERROR/3 (<stdin>, line 284); backlink

Unknown interpreted text role "setting".

Delaying a scope programmatically

You can delay a :ref:`throttling scope <throttling-scopes>` on demand through :meth:`crawler.throttler.delay_scope() <scrapy.throttling.ThrottlingManagerProtocol.delay_scope>`:

System Message: ERROR/3 (<stdin>, line 293); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 293); backlink

Unknown interpreted text role "meth".

System Message: WARNING/2 (<stdin>, line 297)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    crawler.throttler.delay_scope("example.com", 30.0)

This holds back every request of the scope for at least the given number of seconds, counted as a :ref:`backoff <backoff>` trigger.

System Message: ERROR/3 (<stdin>, line 301); backlink

Unknown interpreted text role "ref".

It is useful to react to situations that :ref:`automatic backoff <backoff>` cannot detect on its own, such as a soft block that comes back as a 200 response. For example, a spider callback can slow down the whole domain when it detects a maintenance page, and reschedule the current request:

System Message: ERROR/3 (<stdin>, line 304); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 309)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy import Request, Spider
    from scrapy.utils.httpobj import urlparse_cached


    class MySpider(Spider):
        name = "myspider"
        start_urls = ["https://example.com/"]

        def parse(self, response):
            if "under maintenance" in response.text:
                scope = urlparse_cached(response).netloc
                self.crawler.throttler.delay_scope(scope, 600.0)
                yield response.request.replace(dont_filter=True)
                return
            # Normal parsing follows.

Unlike :ref:`untrusted delays <rate-limiting-headers>`, this delay is not capped at :setting:`BACKOFF_MAX_DELAY`.

System Message: ERROR/3 (<stdin>, line 327); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 327); backlink

Unknown interpreted text role "setting".

Per-request throttling

Sometimes you need different throttling behavior for individual requests or for request groups that are not tied to a specific domain.

For example, you might want to throttle API endpoints differently than web pages on the same domain, group requests by content type (images vs HTML), or apply different throttling based on request priority.

System Message: ERROR/3 (<stdin>, line 342)

Unknown directive type "reqmeta".

.. reqmeta:: throttling_scopes

Use the throttling_scopes request metadata to assign requests to custom throttling groups:

System Message: WARNING/2 (<stdin>, line 351)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://api.example/", meta={"throttling_scopes": "api"})

You can also assign multiple throttling groups to a single request:

System Message: WARNING/2 (<stdin>, line 357)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://api.example/users", meta={"throttling_scopes": {"api", "users"}})

You can then use the :setting:`THROTTLING_SCOPES` setting to customize throttling for such requests:

System Message: ERROR/3 (<stdin>, line 361); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 364)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api": {"concurrency": 2},
        "users": {"delay": 5.0},
    }

Note

These custom throttling groups persist through redirects. For redirect-aware throttling assignment, see :ref:`custom-throttling-scopes`.

System Message: ERROR/3 (<stdin>, line 372); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 375)

Unknown directive type "reqmeta".

.. reqmeta:: throttling_delay

Delaying a single request

To hold a single request for a fixed number of seconds before it is sent, regardless of its scopes, set the throttling_delay request metadata key:

System Message: WARNING/2 (<stdin>, line 383)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://example.com/slow", meta={"throttling_delay": 5.0})

The delay is applied once, the first time the request reaches the throttling gate.

throttling_delay defines only the earliest time the request may be sent, not the exact time: once the delay elapses, the request still competes with every other pending request for its scopes. If you want it sent as soon as its delay elapses, give it a higher :attr:`~scrapy.Request.priority` too:

System Message: ERROR/3 (<stdin>, line 390); backlink

Unknown interpreted text role "attr".

System Message: WARNING/2 (<stdin>, line 395)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://example.com/slow", meta={"throttling_delay": 5.0}, priority=1)

Without a higher priority, a backlog of requests ahead of it in a FIFO queue could keep it waiting well past the configured delay; a higher priority puts it at the front of the queue, so it goes out right after its delay.

System Message: ERROR/3 (<stdin>, line 403)

Unknown directive type "reqmeta".

.. reqmeta:: throttling_dont_track

Excluding a request from throttling state

Some requests (authentication flows, one-off API calls, file downloads) should not influence throttling state even if they get a :setting:`BACKOFF_HTTP_CODES` response or raise a :setting:`BACKOFF_EXCEPTIONS` exception. Set the throttling_dont_track request metadata key to True to process such a request normally without letting its outcome trigger :ref:`backoff <backoff>`:

System Message: ERROR/3 (<stdin>, line 408); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 408); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 408); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 414)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://example.com/login", meta={"throttling_dont_track": True})

Throttling scopes

Throttling scopes represent aspects of requests that can be throttled independently.

Customizing throttling scopes

There are 2 ways to customize throttling scopes.

To configure existing scopes, use the :setting:`THROTTLING_SCOPES` setting. Its keys are scope names and its values are :class:`~scrapy.throttling.ThrottlingScopeConfig` dicts, which accept the following keys:

System Message: ERROR/3 (<stdin>, line 437); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 437); backlink

Unknown interpreted text role "class".

concurrency (:class:`int`): System Message: ERROR/3 (<stdin>, line 446); backlink
Unknown interpreted text role "class".

Maximum number of concurrent requests for the scope. When unset, the default depends on the kind of scope: :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` for domain scopes, :setting:`CONCURRENT_REQUESTS_PER_IP` for IP scopes, and :setting:`THROTTLING_SCOPE_CONCURRENCY` for any other scope.

System Message: ERROR/3 (<stdin>, line 443); backlink
Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 443); backlink
Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 443); backlink
Unknown interpreted text role "setting".
min_concurrency (:class:`int`): System Message: ERROR/3 (<stdin>, line 450); backlink
Unknown interpreted text role "class".

Concurrency floor that :ref:`backoff <backoff>` and :ref:`rampup <rampup>` never drop below. Defaults to 1.

System Message: ERROR/3 (<stdin>, line 449); backlink
Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 449); backlink
Unknown interpreted text role "ref".
delay (:class:`float`): System Message: ERROR/3 (<stdin>, line 454); backlink
Unknown interpreted text role "class".

Minimum seconds between requests for the scope. Defaults to :setting:`DOWNLOAD_DELAY`.

System Message: ERROR/3 (<stdin>, line 453); backlink
Unknown interpreted text role "setting".
jitter (:class:`float` or 2-:class:`list`): System Message: ERROR/3 (<stdin>, line 460); backlink
Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 460); backlink
Unknown interpreted text role "class".

Magnitude of the random variation applied to delay; the per-scope override of :setting:`RANDOMIZE_DOWNLOAD_DELAY`. 0 disables it, 0.5 means ±50% (the default when :setting:`RANDOMIZE_DOWNLOAD_DELAY` is on), and a [low, high] pair multiplies the delay by 1 + uniform(low, high).

System Message: ERROR/3 (<stdin>, line 457); backlink
Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 457); backlink
Unknown interpreted text role "setting".
quota (:class:`float`): System Message: ERROR/3 (<stdin>, line 463); backlink
Unknown interpreted text role "class".

Maximum :ref:`quota <throttling-quotas>` consumed per window.

System Message: ERROR/3 (<stdin>, line 463); backlink
Unknown interpreted text role "ref".
window (:class:`float`): System Message: ERROR/3 (<stdin>, line 466); backlink
Unknown interpreted text role "class".

Quota window in seconds. Defaults to :setting:`THROTTLING_WINDOW`.

System Message: ERROR/3 (<stdin>, line 466); backlink
Unknown interpreted text role "setting".
rampup (:class:`bool` or :class:`dict`): System Message: ERROR/3 (<stdin>, line 469); backlink
Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 469); backlink
Unknown interpreted text role "class".

Enables :ref:`rampup <rampup>` for the scope.

System Message: ERROR/3 (<stdin>, line 469); backlink
Unknown interpreted text role "ref".
backoff (:class:`~scrapy.throttling.BackoffConfig`): System Message: ERROR/3 (<stdin>, line 472); backlink
Unknown interpreted text role "class".

Per-scope :ref:`backoff overrides <per-scope-backoff>`.

System Message: ERROR/3 (<stdin>, line 472); backlink
Unknown interpreted text role "ref".
manager (:class:`str` or :class:`type`): System Message: ERROR/3 (<stdin>, line 476); backlink
Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 476); backlink
Unknown interpreted text role "class".

Import path or class of a :ref:`custom scope manager <custom-throttling-scope-managers>` for the scope.

System Message: ERROR/3 (<stdin>, line 475); backlink
Unknown interpreted text role "ref".
ignore_robots_txt (:class:`bool`): System Message: ERROR/3 (<stdin>, line 480); backlink
Unknown interpreted text role "class".

Silences the warning logged when this configuration is more aggressive than a :ref:`robots.txt Crawl-delay <crawl-delay>`.

System Message: ERROR/3 (<stdin>, line 479); backlink
Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 482)

Unknown directive type "setting".

.. setting:: THROTTLING_MANAGER

To change how scopes are assigned (or anything beyond per-scope settings), set :setting:`THROTTLING_MANAGER` (default: :class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component <topics-components>` that implements the :class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import path as a string):

System Message: ERROR/3 (<stdin>, line 484); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 484); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 484); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 484); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 491)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager"

Handling of multiple throttling scopes

When a request has multiple throttling scopes, it is not sent until all of its throttling scopes allow it.

Throttling quotas

When different requests can consume different amounts of a throttling scope, you can express this using throttling quotas.

System Message: ERROR/3 (<stdin>, line 512)

Unknown directive type "setting".

.. setting:: THROTTLING_WINDOW

Use the :setting:`THROTTLING_WINDOW` setting (default: 60.0) or the "window" key in the :setting:`THROTTLING_SCOPES` setting to define the time window after which throttling quotas are reset.

System Message: ERROR/3 (<stdin>, line 514); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 514); backlink

Unknown interpreted text role "setting".

Then use the :setting:`THROTTLING_SCOPES` setting to define the throttling quotas for each throttling scope:

System Message: ERROR/3 (<stdin>, line 518); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 521)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "quota": 500.0,
        },
    }

Then, in the :reqmeta:`throttling_scopes` request metadata key or in the return value of the :meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes` method, define a :class:`dict` where keys are throttling scopes and values are :class:`float` values that indicate the expected quota consumption (it does not need to be exact).

System Message: ERROR/3 (<stdin>, line 530); backlink

Unknown interpreted text role "reqmeta".

System Message: ERROR/3 (<stdin>, line 530); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 530); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 530); backlink

Unknown interpreted text role "class".

Everything else being equal, :class:`~scrapy.pqueues.ScrapyPriorityQueue` will prioritize requests that consume a higher portion of the available throttling quota, to minimize the risk of those requests getting stuck.

System Message: ERROR/3 (<stdin>, line 536); backlink

Unknown interpreted text role "class".

Customizing throttling scope managers

System Message: ERROR/3 (<stdin>, line 545)

Unknown directive type "setting".

.. setting:: THROTTLING_SCOPE_MANAGER

The :setting:`THROTTLING_SCOPE_MANAGER` setting (default: :class:`~scrapy.throttling.ThrottlingScopeManager`) is a :ref:`component <topics-components>` that implements the :class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` (or its import path as a string):

System Message: ERROR/3 (<stdin>, line 547); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 547); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 547); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 547); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 553)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPE_MANAGER = "myproject.throttling.MyThrottlingScopeManager"

For each throttling scope, an instance of this class is created to manage any gradual :ref:`backoff <backoff>` or :ref:`rampup <rampup>` required at run time.

System Message: ERROR/3 (<stdin>, line 558); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 558); backlink

Unknown interpreted text role "ref".

You can implement your own throttling scope manager if you wish to change the backoff or rampup behavior beyond what settings allow.

You can also define a custom throttling scope manager for a specific throttling scope by setting the "manager" key in the :setting:`THROTTLING_SCOPES` setting:

System Message: ERROR/3 (<stdin>, line 565); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 569)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "manager": "myproject.throttling.MyThrottlingScopeManager",
        },
    }

A scope manager only needs to implement the :class:`~scrapy.throttling.ThrottlingScopeManagerProtocol`. For example, the following manager enforces a fixed quota per time window without any delay or exponential backoff, similar to a fixed-window rate limiter:

System Message: ERROR/3 (<stdin>, line 578); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 583)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``myproject/throttling.py``

    import time


    class FixedWindowScopeManager:
        def __init__(self, crawler, config):
            self.limit = config["quota"]
            self.window = config.get("window", 60.0)
            self.reset_at = None
            self.used = 0.0
            self.active = 0

        @classmethod
        def from_crawler(cls, crawler, config):
            return cls(crawler, config)

        def can_send(self, now=None, amount=None):
            now = now if now is not None else time.monotonic()
            if self.reset_at is not None and now >= self.reset_at:
                self.reset_at, self.used = None, 0.0
            if self.used and self.used + (amount or 0) > self.limit:
                return self.reset_at - now
            return 0.0

        def record_sent(self, now=None, amount=None):
            now = now if now is not None else time.monotonic()
            if self.reset_at is None:
                self.reset_at = now + self.window
            self.used += amount or 0
            self.active += 1

        def record_done(self, now=None):
            self.active = max(0, self.active - 1)

        def record_backoff(self, delay=None, now=None):
            pass  # this manager does not back off

        def reconcile_quota(self, consumed=None, remaining=None, now=None):
            if remaining is not None:
                self.used = max(0.0, self.limit - remaining)
            elif consumed is not None:
                self.used = max(0.0, self.used + consumed)

        def set_base_delay(self, delay):
            pass

        def set_concurrency(self, concurrency):
            pass

        def is_idle(self, now, max_idle):
            return self.active == 0

Throttling-aware scheduling

By default, throttling is enforced at the engine, where a request waiting on its :ref:`throttling scopes <throttling-scopes>` holds a concurrency slot. In a crawl that mixes heavily-throttled scopes with unthrottled ones, this can let throttled requests starve unthrottled ones that could be sent right away (head-of-line blocking; Scrapy logs a warning, see :setting:`DELAYED_REQUESTS_WARN_THRESHOLD`).

System Message: ERROR/3 (<stdin>, line 642); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 642); backlink

Unknown interpreted text role "setting".

:class:`~scrapy.core.scheduler.ThrottlingAwareScheduler` avoids this. To enable it:

System Message: ERROR/3 (<stdin>, line 649); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 652)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    SCHEDULER = "scrapy.core.scheduler.ThrottlingAwareScheduler"
    SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.ThrottlingAwarePriorityQueue"

System Message: ERROR/3 (<stdin>, line 658)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.scheduler.ThrottlingAwareScheduler

Examples

Alternative domain throttling

If you are not happy with the :ref:`default throttling scope behavior <basic-throttling>` with regards to domains and subdomains, you can change it.

System Message: ERROR/3 (<stdin>, line 670); backlink

Unknown interpreted text role "ref".

Alternative approaches include:

Using the highest-level registrable domain as the throttling scope, e.g. https://books.toscrape.com and https://toscrape.com both get a toscrape.com throttling scope.

This allows to apply the same throttling settings to all subdomains of a registrable domain.

For example:

System Message: ERROR/3 (<stdin>, line 684)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    import tldextract
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager:

        def get_request_scopes(self, request):
            extracted = tldextract.extract(request.url)
            if extracted.domain and extracted.suffix:
                return f"{extracted.domain}.{extracted.suffix}"
            return urlparse_cached(request).netloc


    THROTTLING_MANAGER = MyThrottlingManager

Using multiple throttling scopes per request, one per registrable domain and for every higher-level subdomain, e.g. https://books.toscrape.com and https://toscrape.com both get a toscrape.com throttling scope, but https://books.toscrape.com also gets a books.toscrape.com throttling scope.

This allows to apply the same throttling settings to all subdomains of a registrable domain, but also allows applying further restrictions on each or on some subdomains.

For example:

System Message: ERROR/3 (<stdin>, line 714)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    import tldextract
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager:

        def get_request_scopes(self, request):
            extracted = tldextract.extract(request.url)
            if not (extracted.domain and extracted.suffix):
                return urlparse_cached(request).netloc
            scopes = set()
            registrable_domain = f"{extracted.domain}.{extracted.suffix}"
            scopes.add(registrable_domain)
            if extracted.subdomain:
                subdomain_parts = extracted.subdomain.split(".")
                for i in range(len(subdomain_parts)):
                    subdomain = ".".join(subdomain_parts[i:])
                    full_domain = f"{subdomain}.{registrable_domain}"
                    scopes.add(full_domain)
            return scopes


    THROTTLING_MANAGER = MyThrottlingManager
    THROTTLING_SCOPES = {
        "toscrape.com": {"concurrency": 32},
        "books.toscrape.com": {"concurrency": 24},
        "quotes.toscrape.com": {"concurrency": 16},
    }

Here books.toscrape.com requests can reach 24 concurrency and quotes.toscrape.com requests can reach 16 concurrency, but never both at the same time, because that would sum 40 concurrency, and toscrape.com requests are limited to 32.

Endpoint-specific throttling

To apply different throttling settings to different endpoints of the same domain and not enforce any common throttling, effectively treating them as different domains:

Implement a :ref:`throttling manager <custom-throttling-scopes>` that sets endpoint-specific throttling scopes for that domain:

System Message: ERROR/3 (<stdin>, line 760); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 763)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.throttling import ThrottlingManager, scope_cache
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            parsed_url = urlparse_cached(request)
            if parsed_url.netloc != "api.toscrape.com":
                return await super().get_scopes(request)
            return f"{parsed_url.netloc}{parsed_url.path}"

Use the :setting:`THROTTLING_SCOPES` setting to set different throttling settings per endpoint:

System Message: ERROR/3 (<stdin>, line 777); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 780)

Error in "code-block" directive: unknown option: "caption".
```
.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com/fast-endpoint": {"concurrency": 1000, "delay": 0.08},
        "api.toscrape.com/slow-endpoint": {"delay": 5.0},
    }
```

Web scraping API throttling

Imagine you are sending requests to a web scraping API, e.g. to avoid bans. Unless that API provides a Scrapy plugin to make it easier to use, you may want to:

Use the :setting:`THROTTLING_SCOPES` setting to increase concurrency for API requests. For example:

System Message: ERROR/3 (<stdin>, line 798); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 801)

Error in "code-block" directive: unknown option: "caption".
```
.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {"concurrency": 1000, "delay": 0.08},
    }
```

Implement a :ref:`throttling manager <custom-throttling-scopes>` that:

System Message: ERROR/3 (<stdin>, line 808); backlink

Unknown interpreted text role "ref".

Adds a throttling scope for the URL being scraped.

For example, if you request https://api.toscrape.com/?url=https://example.com, by default it will get a api.toscrape.com throttling scope, but it should also get the example.com throttling scope:

System Message: WARNING/2 (<stdin>, line 817)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from urllib.parse import urlparse

    from scrapy.throttling import add_scope, ThrottlingManager, scope_cache
    from scrapy.utils.httpobj import urlparse_cached
    from w3lib.url import url_query_parameter


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            scopes = await super().get_scopes(request)
            if urlparse_cached(request).netloc != "api.toscrape.com":
                return scopes
            target_url = url_query_parameter(request.url, "url")
            if not target_url:
                return scopes
            target_domain = urlparse(target_url).netloc
            return add_scope(scopes, target_domain)

Can differentiate between exhaustion of the target website and exhaustion of the API itself. For example:

System Message: WARNING/2 (<stdin>, line 841)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.throttling import ThrottlingManager
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager(ThrottlingManager):
        async def get_response_backoff(self, response):
            if (
                urlparse_cached(response.request).netloc != "api.toscrape.com"
                or response.status != 200
            ):
                return await super().get_response_backoff(response)
            upstream_status_code = int(
                response.headers.get("X-Upstream-Status-Code", b"200")
            )
            upstream_response = response.__class__(
                response.url,
                status=upstream_status_code,
                headers=response.headers,
                body=response.body,
            )
            return await super().get_response_backoff(upstream_response)

Cost-capped throttling

Imagine you are using an API that charges different requests differently, e.g. based on the features used, and you want to limit how much you spend per time window (:setting:`BACKOFF_WINDOW`). You can use :ref:`throttling quotas <throttling-quotas>` for that:

System Message: ERROR/3 (<stdin>, line 871); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 871); backlink

Unknown interpreted text role "ref".

Implement a :ref:`throttling manager <custom-throttling-scopes>` that:

System Message: ERROR/3 (<stdin>, line 876); backlink

Unknown interpreted text role "ref".

Sets a cost throttling scope on each request to some estimation based e.g. on request URL parameters:

System Message: WARNING/2 (<stdin>, line 881)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.utils.httpobj import urlparse_cached
    from scrapy.throttling import ThrottlingManager, scope_cache


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            scopes = await super().get_scopes(request)
            parsed_url = urlparse_cached(request)
            if parsed_url.netloc != "api.toscrape.com":
                return scopes
            return add_scope(scopes, "cost", estimate_request_cost(request))

Reconciles the estimated cost with the actual cost reported by the response, so that the quota tracks real spending:

System Message: WARNING/2 (<stdin>, line 899)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.throttling import ThrottlingManager, update_scope_backoff


    class MyThrottlingManager(ThrottlingManager):
        async def get_response_backoff(self, response):
            backoff = await super().get_response_backoff(response)
            if response.headers.get("X-Actual-Cost") is None:
                return backoff
            estimated = estimate_request_cost(response.request)
            actual = float(response.headers[b"X-Actual-Cost"])
            # Report the difference between actual and estimated cost.
            return update_scope_backoff(backoff, "cost", consumed=actual - estimated)

Use the :setting:`THROTTLING_SCOPES` setting to set a maximum cost per time window:

System Message: ERROR/3 (<stdin>, line 914); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 917)

Error in "code-block" directive: unknown option: "caption".
```
.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "cost": {"quota": 100.0},
    }
```
This will allow you to spend up to 100.0 units of cost per time window (default: 60 seconds) before throttling kicks in.

Per-IP concurrency limiting

System Message: ERROR/3 (<stdin>, line 932)

Unknown directive type "setting".

.. setting:: CONCURRENT_REQUESTS_PER_IP

A concurrency limit keyed by IP is a throttling scope whose id is the request's IP. A request then carries two scopes, its domain and its IP, and is only sent when both allow it (see :ref:`multiple-throttling-scopes`).

System Message: ERROR/3 (<stdin>, line 934); backlink

Unknown interpreted text role "ref".

Set :setting:`CONCURRENT_REQUESTS_PER_IP` (default: 0, disabled) to a positive number to enable this built in: the throttler adds the IP of each request, resolved from the DNS cache, as a second scope, limited to that many concurrent requests. IP scopes whose concurrency is left unset default to :setting:`CONCURRENT_REQUESTS_PER_IP`.

System Message: ERROR/3 (<stdin>, line 938); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 938); backlink

Unknown interpreted text role "setting".

For finer control (e.g. resolving IPs that are not in the DNS cache yet, or grouping several hosts under one address), implement a :ref:`throttling manager <custom-throttling-scopes>` that adds the request's IP as a second scope instead:

System Message: ERROR/3 (<stdin>, line 944); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 949)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import socket

    from scrapy.throttling import ThrottlingManager, add_scope, scope_cache
    from scrapy.utils.asyncio import run_in_thread
    from scrapy.utils.httpobj import urlparse_cached


    class IPThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            scopes = await super().get_scopes(request)
            host = urlparse_cached(request).hostname
            address = await run_in_thread(socket.gethostbyname, host)
            return add_scope(scopes, address)

Additional settings

System Message: ERROR/3 (<stdin>, line 970)

Unknown directive type "setting".
```
.. setting:: BACKOFF_EXCEPTIONS
```
:setting:`BACKOFF_EXCEPTIONS`

System Message: ERROR/3 (<stdin>, line 972); backlink

Unknown interpreted text role "setting".

Default:
- :exc:`~scrapy.exceptions.DownloadFailedError`
  
  System Message: ERROR/3 (<stdin>, line 976); backlink
  
  Unknown interpreted text role "exc".
- :exc:`~scrapy.exceptions.DownloadTimeoutError`
  
  System Message: ERROR/3 (<stdin>, line 977); backlink
  
  Unknown interpreted text role "exc".
- :exc:`~scrapy.exceptions.ResponseDataLossError`
  
  System Message: ERROR/3 (<stdin>, line 978); backlink
  
  Unknown interpreted text role "exc".
List of exception classes that trigger backoff when raised while downloading a request. Strings are interpreted as import paths.
System Message: ERROR/3 (<stdin>, line 983)

Unknown directive type "seealso".
```
.. seealso:: :setting:`RETRY_EXCEPTIONS`
```
System Message: ERROR/3 (<stdin>, line 985)

Unknown directive type "setting".
```
.. setting:: BACKOFF_JITTER
```
:setting:`BACKOFF_JITTER` (default: 0.1)

System Message: ERROR/3 (<stdin>, line 987); backlink

Unknown interpreted text role "setting".

Random jitter applied to each backoff delay, as a fraction of the delay. With the default value of 0.1 the delay is randomized by ±10%. Overrides :setting:`RANDOMIZE_DOWNLOAD_DELAY` during backoff.

System Message: ERROR/3 (<stdin>, line 989); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 993)

Unknown directive type "setting".
```
.. setting:: BACKOFF_MIN_DELAY
```
:setting:`BACKOFF_MIN_DELAY` (default: 1.0)

System Message: ERROR/3 (<stdin>, line 995); backlink

Unknown interpreted text role "setting".

Delay, in seconds, applied on the first backoff step (and the minimum delay during backoff).
System Message: ERROR/3 (<stdin>, line 1000)

Unknown directive type "setting".
```
.. setting:: BACKOFF_WINDOW
```
:setting:`BACKOFF_WINDOW` (default: 60.0)

System Message: ERROR/3 (<stdin>, line 1002); backlink

Unknown interpreted text role "setting".

Time window, in seconds, used by :ref:`backoff <backoff>` and :ref:`rampup <rampup>`. During backoff, a :ref:`throttling scope <throttling-scopes>` must go this many seconds without a new backoff trigger (an HTTP error code from :setting:`BACKOFF_HTTP_CODES` or an exception from :setting:`BACKOFF_EXCEPTIONS`) before its delay decreases by one :setting:`BACKOFF_DELAY_FACTOR` step toward the configured value. A new trigger resets the countdown.

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1004); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 1012)

Unknown directive type "setting".
```
.. setting:: DELAYED_REQUESTS_WARN_THRESHOLD
```
:setting:`DELAYED_REQUESTS_WARN_THRESHOLD` (default: 500)

System Message: ERROR/3 (<stdin>, line 1014); backlink

Unknown interpreted text role "setting".

Number of requests held back by :ref:`throttling <throttling>` at which Scrapy logs a warning, to help detect throttling configurations that hold back more requests than expected.

System Message: ERROR/3 (<stdin>, line 1016); backlink

Unknown interpreted text role "ref".

While throttled, requests in the :ref:`scheduler <topics-scheduler>` remain in the scheduler. However, requests sent with :meth:`engine.download() <scrapy.core.engine.ExecutionEngine.download>` bypass the scheduler, including requests sent by some built-in :ref:`components <topics-components>` and :ref:`inline requests <inline-requests>`. When such requests are throttled, they are paused and kept in memory, along with any run time context from the code that is sending them. If they accumulate, they can become a memory issue.

System Message: ERROR/3 (<stdin>, line 1020); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1020); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 1020); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1020); backlink

Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 1029)

Unknown directive type "setting".
```
.. setting:: RANDOMIZE_DOWNLOAD_DELAY
```
:setting:`RANDOMIZE_DOWNLOAD_DELAY` (default: True)

System Message: ERROR/3 (<stdin>, line 1031); backlink

Unknown interpreted text role "setting".

Randomize delays by this factor, e.g. if 0.2 randomize delays between delay*0.8 and delay*1.2.

It can be set to a 2-item list with low and high factors, e.g. [-0.1, 0.3] to randomize delays between delay*0.9 and delay*1.3.

If True, 0.5 (i.e. ±50%) is used as the randomization factor. If False, no randomization is applied.
System Message: ERROR/3 (<stdin>, line 1043)

Unknown directive type "setting".
```
.. setting:: THROTTLING_DEBUG
```
:setting:`THROTTLING_DEBUG` (default: False)

System Message: ERROR/3 (<stdin>, line 1045); backlink

Unknown interpreted text role "setting".

Whether to log :ref:`throttling <throttling>` decisions (per-scope delays, backoff steps and recoveries) for debugging.

System Message: ERROR/3 (<stdin>, line 1047); backlink

Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 1050)

Unknown directive type "setting".
```
.. setting:: THROTTLING_SCOPE_CONCURRENCY
```
:setting:`THROTTLING_SCOPE_CONCURRENCY` (default: 1)

System Message: ERROR/3 (<stdin>, line 1052); backlink

Unknown interpreted text role "setting".

Default maximum number of concurrent requests for :ref:`throttling scopes <throttling-scopes>` that are neither domains nor IPs, e.g. custom scopes added by a :ref:`custom throttling manager <custom-throttling-scopes>`.

System Message: ERROR/3 (<stdin>, line 1054); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1054); backlink

Unknown interpreted text role "ref".

Domain and IP scopes use :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`CONCURRENT_REQUESTS_PER_IP` instead. A scope concurrency set in :setting:`THROTTLING_SCOPES` overrides this.

System Message: ERROR/3 (<stdin>, line 1058); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1058); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1058); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 1062)

Unknown directive type "setting".
```
.. setting:: THROTTLING_SCOPE_LIMIT
```
:setting:`THROTTLING_SCOPE_LIMIT` (default: 100000)

System Message: ERROR/3 (<stdin>, line 1064); backlink

Unknown interpreted text role "setting".

Maximum number of :ref:`throttling scope <throttling-scopes>` states kept in memory at once, to bound memory usage on broad crawls that touch a large number of scopes (e.g. domains).

System Message: ERROR/3 (<stdin>, line 1066); backlink

Unknown interpreted text role "ref".

When the limit is exceeded, the least-recently-used idle scopes are evicted (an evicted scope is recreated from its configuration the next time it is needed). Scopes with in-flight requests or in active backoff are never evicted, so the limit may be temporarily exceeded if that many scopes are busy at once. Set to 0 to disable the limit.

This complements :setting:`THROTTLING_SCOPE_MAX_IDLE`, which evicts scopes by inactivity time rather than by count.

System Message: ERROR/3 (<stdin>, line 1076); backlink

Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 1079)

Unknown directive type "setting".
```
.. setting:: THROTTLING_SCOPE_MAX_IDLE
```
:setting:`THROTTLING_SCOPE_MAX_IDLE` (default: 3600.0)

System Message: ERROR/3 (<stdin>, line 1081); backlink

Unknown interpreted text role "setting".

Seconds of inactivity after which the state of a :ref:`throttling scope <throttling-scopes>` is evicted from memory to bound memory usage on long-running crawls. Set to 0 to never evict. Scopes in active backoff are never evicted.

System Message: ERROR/3 (<stdin>, line 1083); backlink

Unknown interpreted text role "ref".

Migrating from AutoThrottle

The AutoThrottle extension is deprecated in favor of the throttling and :ref:`backoff <backoff>` system described here, which is always active and does not need to be enabled.

System Message: ERROR/3 (<stdin>, line 1093); backlink

Unknown interpreted text role "ref".

Setting AUTOTHROTTLE_ENABLED to True still works but logs a deprecation warning. To migrate, drop the AUTOTHROTTLE_* settings and use the following equivalents:

AutoThrottle	Throttling
`AUTOTHROTTLE_ENABLED = True`	No equivalent; throttling is always active.
`AUTOTHROTTLE_START_DELAY`	:setting:`DOWNLOAD_DELAY` System Message: ERROR/3 (`<stdin>`, line 1109); backlink Unknown interpreted text role "setting".
`AUTOTHROTTLE_MAX_DELAY`	:setting:`BACKOFF_MAX_DELAY` System Message: ERROR/3 (`<stdin>`, line 1111); backlink Unknown interpreted text role "setting".
`AUTOTHROTTLE_TARGET_CONCURRENCY`	:ref:`rampup <rampup>` (`"rampup": True`) System Message: ERROR/3 (`<stdin>`, line 1113); backlink Unknown interpreted text role "ref".
`AUTOTHROTTLE_DEBUG = True`	:setting:`THROTTLING_DEBUG` `= True` System Message: ERROR/3 (`<stdin>`, line 1115); backlink Unknown interpreted text role "setting".
`autothrottle_dont_adjust_delay` (request meta)	:reqmeta:`throttling_dont_track` (request meta) System Message: ERROR/3 (`<stdin>`, line 1117); backlink Unknown interpreted text role "reqmeta".

AutoThrottle adjusted the delay of each download slot based on response latency. The new system does not measure latency; instead, it reacts to explicit rate-limit signals (:setting:`BACKOFF_HTTP_CODES`, :setting:`BACKOFF_EXCEPTIONS`, :ref:`Retry-After / RateLimit-Reset <rate-limiting-headers>`) and, with :ref:`rampup <rampup>`, probes for the fastest rate a scope tolerates. If you specifically need latency-based control, implement a custom :ref:`throttling scope manager <custom-throttling-scope-managers>`.

System Message: ERROR/3 (<stdin>, line 1119); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1119); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 1119); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1119); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 1119); backlink

Unknown interpreted text role "ref".

API

System Message: ERROR/3 (<stdin>, line 1134)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol
    :members:
    :member-order: bysource

System Message: ERROR/3 (<stdin>, line 1138)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingManager
    :members: get_response_delay

System Message: ERROR/3 (<stdin>, line 1141)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingScopeManagerProtocol
    :members:
    :member-order: bysource

System Message: ERROR/3 (<stdin>, line 1145)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingScopeManager

System Message: ERROR/3 (<stdin>, line 1147)

Unknown directive type "autoclass".

.. autoclass:: scrapy.pqueues.ThrottlingAwarePriorityQueue

System Message: ERROR/3 (<stdin>, line 1149)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingScopeConfig

System Message: ERROR/3 (<stdin>, line 1151)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.BackoffConfig

System Message: ERROR/3 (<stdin>, line 1153)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.scope_cache

System Message: ERROR/3 (<stdin>, line 1154)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.add_scope

System Message: ERROR/3 (<stdin>, line 1155)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.update_scope_backoff

</html>

40 KiB Raw Blame History Unescape Escape

Throttling

Concurrency and delay

Per-domain throttling

Backoff

How backoff works

Per-scope backoff configuration

Rampup

Rate limiting headers

robots.txt

Delaying a scope programmatically

Per-request throttling

Delaying a single request

Excluding a request from throttling state

Throttling scopes

Customizing throttling scopes

Handling of multiple throttling scopes

Throttling quotas

Customizing throttling scope managers

Throttling-aware scheduling

Examples

Alternative domain throttling

Endpoint-specific throttling

Web scraping API throttling

Cost-capped throttling

Per-IP concurrency limiting

Additional settings

Migrating from AutoThrottle

API

40 KiB

Raw Blame History