RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/throttling.rst

35 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Throttling

Sending too many requests too quickly can overwhelm websites. :ref:`Throttling <basic-throttling>` and :ref:`backoff <backoff>` aim to prevent that.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Concurrency and delay

Requests are throttled on a per-domain basis by default [1]. This allows efficient crawling of multiple sites simultaneously.

Each domain and subdomain is treated separately: requests to books.toscrape.com and quotes.toscrape.com each have their own throttling limits, as do toscrape.com and books.toscrape.com.

The main throttling :ref:`settings <topics-settings>` are:

System Message: ERROR/3 (<stdin>, line 25); backlink

Unknown interpreted text role "ref".

  • System Message: ERROR/3 (<stdin>, line 27)

    Unknown directive type "setting".

    .. setting:: CONCURRENT_REQUESTS_PER_DOMAIN

    :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (default: 1 (:ref:`fallback <default-settings>`: 8))

    System Message: ERROR/3 (<stdin>, line 29); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 29); backlink

    Unknown interpreted text role "ref".

    Maximum number of simultaneous requests per domain.

    It defines a number of “slots” per domain. Each slot can send 1 request at a time: it sends a request, waits for the response, then sends the next request, and so on.

  • System Message: ERROR/3 (<stdin>, line 37)

    Unknown directive type "setting".

    .. setting:: DOWNLOAD_DELAY

    :setting:`DOWNLOAD_DELAY` (default: 1 (:ref:`fallback <default-settings>`: 0))

    System Message: ERROR/3 (<stdin>, line 39); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 39); backlink

    Unknown interpreted text role "ref".

    Minimum seconds between any two requests to the same domain.

    Even if you have multiple slots, requests to the same domain cannot be sent more frequently than this delay.

  • System Message: ERROR/3 (<stdin>, line 46)

    Unknown directive type "setting".

    .. setting:: DOWNLOAD_DELAY_PER_SLOT

    :setting:`DOWNLOAD_DELAY_PER_SLOT` (default: 1.0)

    System Message: ERROR/3 (<stdin>, line 48); backlink

    Unknown interpreted text role "setting".

    Minimum seconds between requests in the same slot.

    If a slot sends a request and receives its response before this delay has elapsed, it must wait before sending the next request. The wait time is measured from when the previous request was sent.

For example, with CONCURRENT_REQUESTS_PER_DOMAIN = 2, DOWNLOAD_DELAY = 0.3, and DOWNLOAD_DELAY_PER_SLOT = 1.0, sending 3 requests to the same domain would result in:

T=0.0s: Request 1 sent (slot 1)
T=0.3s: Request 2 sent (slot 2, respects same-domain delay)
T=0.6s: Request 3 must wait (same-domain delay satisfied, but slot 1 needs 1.0s)
T=1.0s: Request 3 sent (slot 1 can now be reused)

When configuring these settings, note that:

  • :setting:`CONCURRENT_REQUESTS` caps CONCURRENT_REQUESTS_PER_DOMAIN.

    System Message: ERROR/3 (<stdin>, line 69); backlink

    Unknown interpreted text role "setting".

  • If DOWNLOAD_DELAY ≥ response time, concurrency is effectively 1. This happens because all slots must wait for the delay between requests, preventing them from sending requests simultaneously.

[1](1, 2)

You can :ref:`customize <throttling-scopes>` how requests are grouped for throttling, but domain-based throttling works well in most cases. For more complex domain grouping strategies, see :ref:`alternative-domain-throttling`.

System Message: ERROR/3 (<stdin>, line 75); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 75); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 81)

Unknown directive type "setting".

.. setting:: THROTTLING_SCOPES

Per-domain throttling

The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling behavior for specific domains [1].

System Message: ERROR/3 (<stdin>, line 87); backlink

Unknown interpreted text role "setting".

Its default value allows faster crawling of the testing website using during the :ref:`tutorial <intro-tutorial>` while maintaining conservative defaults for other domains:

System Message: ERROR/3 (<stdin>, line 90); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 94)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    THROTTLING_SCOPES = {
        "quotes.toscrape.com": {"concurrency": 16, "delay": 0.0},
    }

Additional keys like "jitter" and "backoff" can be used here and are covered later on.

Backoff

When servers respond with rate limiting errors (like HTTP 429) or network timeouts occur, request rate is automatically reduced using exponential backoff.

The key settings are:

  • System Message: ERROR/3 (<stdin>, line 117)

    Unknown directive type "setting".

    .. setting:: BACKOFF_HTTP_CODES

    :setting:`BACKOFF_HTTP_CODES` (default: [429, 502, 503, 504, 520, 521, 522, 523, 524])

    System Message: ERROR/3 (<stdin>, line 119); backlink

    Unknown interpreted text role "setting".

    HTTP status codes that trigger backoff.

  • System Message: ERROR/3 (<stdin>, line 123)

    Unknown directive type "setting".

    .. setting:: BACKOFF_DELAY_FACTOR

    :setting:`BACKOFF_DELAY_FACTOR` (default: 2.0)

    System Message: ERROR/3 (<stdin>, line 125); backlink

    Unknown interpreted text role "setting".

    Each backoff multiplies delay by this factor (2x, 4x, 8x, etc.).

  • System Message: ERROR/3 (<stdin>, line 129)

    Unknown directive type "setting".

    .. setting:: BACKOFF_MAX_DELAY

    :setting:`BACKOFF_MAX_DELAY` (default: 300.0)

    System Message: ERROR/3 (<stdin>, line 131); backlink

    Unknown interpreted text role "setting".

    Maximum delay cap to prevent excessively long waits.

Rampup

When using APIs that charge per request, like web scraping APIs, you often want to maximize throughput while staying within rate limits. To do that, set "rampup" to True in :setting:`THROTTLING_SCOPES`:

System Message: ERROR/3 (<stdin>, line 141); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 145)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "rampup": True,
        },
    }

Rampup increases concurrency or lowers delay as needed based on the following setting:

  • System Message: ERROR/3 (<stdin>, line 157)

    Unknown directive type "setting".

    .. setting:: RAMPUP_BACKOFF_TARGET

    :setting:`RAMPUP_BACKOFF_TARGET` (default: 1)

    System Message: ERROR/3 (<stdin>, line 159); backlink

    Unknown interpreted text role "setting".

    Target number of backoff responses per rampup window, indicating optimal throughput. Can be a range like [1, 3].

Rate limiting headers

Servers may include Retry-After or RateLimit-Reset headers to indicate when you should make your next request. These headers are respected automatically during :ref:`backoff <backoff>`, using their values as minimum delays (capped at :setting:`BACKOFF_MAX_DELAY`).

System Message: ERROR/3 (<stdin>, line 171); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 171); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 179)

Unknown directive type "seealso".

.. seealso:: :setting:`REDIRECT_MAX_DELAY`

robots.txt

Crawl-Delay is a non-standard robots.txt directive that indicates a number of seconds to wait between requests.

System Message: ERROR/3 (<stdin>, line 191)

Unknown directive type "setting".

.. setting:: THROTTLING_ROBOTSTXT_OBEY

System Message: ERROR/3 (<stdin>, line 192)

Unknown directive type "setting".

.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY

If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLING_ROBOTSTXT_OBEY` are True (default), valid Crawl-Delay directives override :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`. Concurrency is set to 1 and delay is set to the value of Crawl-Delay, capped at :setting:`THROTTLING_ROBOTSTXT_MAX_DELAY` (default: 60.0).

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "setting".

If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it will be respected, but a warning will be logged about the discrepancy with Crawl-Delay. Set ignore_robots_txt to True to silence this warning.

System Message: ERROR/3 (<stdin>, line 200); backlink

Unknown interpreted text role "setting".

Per-request throttling

Sometimes you need different throttling behavior for individual requests or for request groups that are not tied to a specific domain.

For example, you might want to throttle API endpoints differently than web pages on the same domain, group requests by content type (images vs HTML), or apply different throttling based on request priority.

System Message: ERROR/3 (<stdin>, line 217)

Unknown directive type "reqmeta".

.. reqmeta:: throttling_scopes

Use the throttling_scopes request metadata to assign requests to custom throttling groups:

System Message: WARNING/2 (<stdin>, line 222)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://api.example/", meta={"throttling_scopes": "api"})

You can also assign multiple throttling groups to a single request:

System Message: WARNING/2 (<stdin>, line 228)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    Request("https://api.example/users", meta={"throttling_scopes": {"api", "users"}})

You can then use the :setting:`THROTTLING_SCOPES` setting to customize throttling for such requests:

System Message: ERROR/3 (<stdin>, line 232); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 235)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api": {"concurrency": 2},
        "users": {"delay": 5.0},
    }

Note

These custom throttling groups persist through redirects. For redirect-aware throttling assignment, see :ref:`custom-throttling-scopes`.

System Message: ERROR/3 (<stdin>, line 243); backlink

Unknown interpreted text role "ref".

Throttling scopes

Throttling scopes represent aspects of requests that can be throttled independently.

Customizing throttling scopes

There are 2 ways to customize throttling scopes.

System Message: ERROR/3 (<stdin>, line 266)

Unknown directive type "setting".

.. setting:: THROTTLING_MANAGER

For anything else, set :setting:`THROTTLING_MANAGER` (default: :class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component <topics-components>` that implements the :class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import path as a string):

System Message: ERROR/3 (<stdin>, line 268); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 268); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 268); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 268); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 274)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager"

Handling of multiple throttling scopes

When a request has multiple throttling scopes, it is not sent until all of its throttling scopes allow it.

Throttling quotas

When different requests can consume different amounts of a throttling scope, you can express this using throttling quotas.

System Message: ERROR/3 (<stdin>, line 297)

Unknown directive type "setting".

.. setting:: THROTTLING_WINDOW

Use the :setting:`THROTTLING_WINDOW` setting (default: 60.0) or the "window" key in the :setting:`THROTTLING_SCOPES` setting to define the time window after which throttling quotas are reset.

System Message: ERROR/3 (<stdin>, line 299); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 299); backlink

Unknown interpreted text role "setting".

Then use the :setting:`THROTTLING_SCOPES` setting to define the throttling quotas for each throttling scope:

System Message: ERROR/3 (<stdin>, line 303); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 306)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "quota": 500.0,
        },
    }

Then, in the :reqmeta:`throttling_scopes` request metadata key or in the return value of the :meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes` method, define a :class:`dict` where keys are throttling scopes and values are :class:`float` values that indicate the expected quota consumption (it does not need to be exact).

System Message: ERROR/3 (<stdin>, line 315); backlink

Unknown interpreted text role "reqmeta".

System Message: ERROR/3 (<stdin>, line 315); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 315); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 315); backlink

Unknown interpreted text role "class".

Everything else being equal, :class:`~scrapy.pqueues.ScrapyPriorityQueue` will prioritize requests that consume a higher portion of the available throttling quota, to minimize the risk of those requests getting stuck.

System Message: ERROR/3 (<stdin>, line 321); backlink

Unknown interpreted text role "class".

Customizing throttling scope managers

System Message: ERROR/3 (<stdin>, line 331)

Unknown directive type "setting".

.. setting:: THROTTLING_SCOPE_MANAGER

The :setting:`THROTTLING_SCOPE_MANAGER` setting (default: :class:`~scrapy.throttling.ThrottlingScopeManager`) is a :ref:`component <topics-components>` that implements the :class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` (or its import path as a string):

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 339)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPE_MANAGER = "myproject.throttling.MyThrottlingScopeManager"

For each throttling scope, an instance of this class is created to manage any gradual :ref:`backoff <backoff>` or :ref:`rampup <rampup>` required at run time.

System Message: ERROR/3 (<stdin>, line 344); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 344); backlink

Unknown interpreted text role "ref".

You can implement your own throttling scope manager if you wish to change the backoff or rampup behavior beyond what settings allow.

You can also define a custom throttling scope manager for a specific throttling scope by setting the "manager" key in the :setting:`THROTTLING_SCOPES` setting:

System Message: ERROR/3 (<stdin>, line 351); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 355)

Error in "code-block" directive: unknown option: "caption".

.. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {
            "manager": "myproject.throttling.MyThrottlingScopeManager",
        },
    }

Examples

Alternative domain throttling

If you are not happy with the :ref:`default throttling scope behavior <basic-throttling>` with regards to domains and subdomains, you can change it.

System Message: ERROR/3 (<stdin>, line 375); backlink

Unknown interpreted text role "ref".

Alternative approaches include:

  • Using the highest-level registrable domain as the throttling scope, e.g. https://books.toscrape.com and https://toscrape.com both get a toscrape.com throttling scope.

    This allows to apply the same throttling settings to all subdomains of a registrable domain.

    For example:

    System Message: ERROR/3 (<stdin>, line 389)

    Error in "code-block" directive: unknown option: "caption".

    .. code-block:: python
    :caption: ``settings.py``

    import tldextract
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager:

        def get_request_scopes(self, request):
            extracted = tldextract.extract(request.url)
            if extracted.domain and extracted.suffix:
                return f"{extracted.domain}.{extracted.suffix}"
            return urlparse_cached(request).netloc


    THROTTLING_MANAGER = MyThrottlingManager

  • Using multiple throttling scopes per request, one per registrable domain and for every higher-level subdomain, e.g. https://books.toscrape.com and https://toscrape.com both get a toscrape.com throttling scope, but https://books.toscrape.com also gets a books.toscrape.com throttling scope.

    This allows to apply the same throttling settings to all subdomains of a registrable domain, but also allows applying further restrictions on each or on some subdomains.

    For example:

    System Message: ERROR/3 (<stdin>, line 419)

    Error in "code-block" directive: unknown option: "caption".

    .. code-block:: python
    :caption: ``settings.py``

    import tldextract
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager:

        def get_request_scopes(self, request):
            extracted = tldextract.extract(request.url)
            if not (extracted.domain and extracted.suffix):
                return urlparse_cached(request).netloc
            scopes = set()
            registrable_domain = f"{extracted.domain}.{extracted.suffix}"
            scopes.add(registrable_domain)
            if extracted.subdomain:
                subdomain_parts = extracted.subdomain.split(".")
                for i in range(len(subdomain_parts)):
                    subdomain = ".".join(subdomain_parts[i:])
                    full_domain = f"{subdomain}.{registrable_domain}"
                    scopes.add(full_domain)
            return scopes


    THROTTLING_MANAGER = MyThrottlingManager
    THROTTLING_SCOPES = {
        "toscrape.com": {"concurrency": 32},
        "books.toscrape.com": {"concurrency": 24},
        "quotes.toscrape.com": {"concurrency": 16},
    }

    Here books.toscrape.com requests can reach 24 concurrency and quotes.toscrape.com requests can reach 16 concurrency, but never both at the same time, because that would sum 40 concurrency, and toscrape.com requests are limited to 32.

Endpoint-specific throttling

To apply different throttling settings to different endpoints of the same domain and not enforce any common throttling, effectively treating them as different domains:

  • Implement a :ref:`throttling manager <custom-throttling-scopes>` that sets endpoint-specific throttling scopes for that domain:

    System Message: ERROR/3 (<stdin>, line 465); backlink

    Unknown interpreted text role "ref".

    System Message: WARNING/2 (<stdin>, line 468)

    Cannot analyze code. Pygments package not found.

    .. code-block:: python

    from scrapy.throttling import ThrottlingManager, scope_cache
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            parsed_url = urlparse_cached(request)
            if parsed_url.netloc != "api.toscrape.com":
                return await super().get_scopes(request)
            return f"{parsed_url.netloc}{parsed_url.path}"

  • Use the :setting:`THROTTLING_SCOPES` setting to set different throttling settings per endpoint:

    System Message: ERROR/3 (<stdin>, line 482); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 485)

    Error in "code-block" directive: unknown option: "caption".

    .. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com/fast-endpoint": {"concurrency": 1000, "delay": 0.08},
        "api.toscrape.com/slow-endpoint": {"delay": 5.0},
    }

Web scraping API throttling

Imagine you are sending requests to a web scraping API, e.g. to avoid bans. Unless that API provides a Scrapy plugin to make it easier to use, you may want to:

  • Use the :setting:`THROTTLING_SCOPES` setting to increase concurrency for API requests. For example:

    System Message: ERROR/3 (<stdin>, line 503); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 506)

    Error in "code-block" directive: unknown option: "caption".

    .. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "api.toscrape.com": {"concurrency": 1000, "delay": 0.08},
    }

  • Implement a :ref:`throttling manager <custom-throttling-scopes>` that:

    System Message: ERROR/3 (<stdin>, line 513); backlink

    Unknown interpreted text role "ref".

    • Adds a throttling scope for the URL being scraped.

      For example, if you request https://api.toscrape.com/?url=https://example.com, by default it will get a api.toscrape.com throttling scope, but it should also get the example.com throttling scope:

      System Message: WARNING/2 (<stdin>, line 522)

      Cannot analyze code. Pygments package not found.

      .. code-block:: python

    from urllib.parse import urlparse

    from scrapy.throttling import add_scope, ThrottlingManager, scope_cache
    from scrapy.utils.httpobj import urlparse_cached
    from w3lib.url import url_query_parameter


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            scopes = await super().get_scopes(request)
            if urlparse_cached(request).netloc != "api.toscrape.com":
                return scopes
            target_url = url_query_parameter(request.url, "url")
            if not target_url:
                return scopes
            target_domain = urlparse(target_url).netloc
            return add_scope(scopes, target_domain)

    • Can differentiate between exhaustion of the target website and exhaustion of the API itself. For example:

      System Message: WARNING/2 (<stdin>, line 546)

      Cannot analyze code. Pygments package not found.

      .. code-block:: python

    from scrapy.throttling import ThrottlingManager
    from scrapy.utils.httpobj import urlparse_cached


    class MyThrottlingManager(ThrottlingManager):
        async def get_response_backoff(self, response):
            if (
                urlparse_cached(response.request).netloc != "api.toscrape.com"
                or response.status != 200
            ):
                return await super().get_response_backoff(response)
            upstream_status_code = int(
                response.headers.get("X-Upstream-Status-Code", b"200")
            )
            upstream_response = response.__class__(
                response.url,
                status=upstream_status_code,
                headers=response.headers,
                body=response.body,
            )
            return await super().get_response_backoff(upstream_response)

Cost-capped throttling

Imagine you are using an API that charges different requests differently, e.g. based on the features used, and you want to limit how much you spend per time window (:setting:`BACKOFF_WINDOW`). You can use :ref:`throttling quotas <throttling-quotas>` for that:

System Message: ERROR/3 (<stdin>, line 576); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 576); backlink

Unknown interpreted text role "ref".

  • Implement a :ref:`throttling manager <custom-throttling-scopes>` that:

    System Message: ERROR/3 (<stdin>, line 581); backlink

    Unknown interpreted text role "ref".

    • Sets a cost throttling scope on each request to some estimation based e.g. on request URL parameters:

      System Message: WARNING/2 (<stdin>, line 586)

      Cannot analyze code. Pygments package not found.

      .. code-block:: python

    from scrapy.utils.httpobj import urlparse_cached
    from scrapy.throttling import ThrottlingManager, scope_cache


    class MyThrottlingManager(ThrottlingManager):
        @scope_cache
        async def get_scopes(self, request):
            scopes = await super().get_scopes(request)
            parsed_url = urlparse_cached(request)
            if parsed_url.netloc != "api.toscrape.com":
                return scopes
            return add_scope(scopes, "cost", estimate_request_cost(request))

    • Reports the actual cost during response parsing:

      System Message: WARNING/2 (<stdin>, line 603)

      Cannot analyze code. Pygments package not found.

      .. code-block:: python

    from scrapy.throttling import ThrottlingManager


    class MyThrottlingManager(ThrottlingManager):
        async def get_response_backoff(self, response):
            scopes = await super().get_response_backoff(response)
            if "cost" not in scopes:
                return scopes
            actual_cost = float(response.headers.get("X-Actual-Cost", b"0"))
            return update_scope_backoff(scopes, "cost", consumed_quota=actual_cost)

  • Use the :setting:`THROTTLING_SCOPES` setting to set a maximum cost per time window:

    System Message: ERROR/3 (<stdin>, line 616); backlink

    Unknown interpreted text role "setting".

    System Message: ERROR/3 (<stdin>, line 619)

    Error in "code-block" directive: unknown option: "caption".

    .. code-block:: python
    :caption: ``settings.py``

    THROTTLING_SCOPES = {
        "cost": {"quota": 100.0},
    }

    This will allow you to spend up to 100.0 units of cost per time window (default: 60 seconds) before throttling kicks in.

Additional settings

  • System Message: ERROR/3 (<stdin>, line 634)

    Unknown directive type "setting".

    .. setting:: BACKOFF_EXCEPTIONS

    :setting:`BACKOFF_EXCEPTIONS`

    System Message: ERROR/3 (<stdin>, line 636); backlink

    Unknown interpreted text role "setting".

    Default:

    System Message: WARNING/2 (<stdin>, line 640)

    Cannot analyze code. Pygments package not found.

    .. code-block:: python

    [
        "twisted.internet.defer.TimeoutError",
        "twisted.internet.error.TimeoutError",
        "twisted.internet.error.TCPTimedOutError",
        "twisted.web.client.ResponseFailed",
    ]

    Exception classes that trigger backoff. Strings are interpreted as import paths.

    System Message: ERROR/3 (<stdin>, line 652)

    Unknown directive type "seealso".

    .. seealso:: :setting:`RETRY_EXCEPTIONS`

  • System Message: ERROR/3 (<stdin>, line 654)

    Unknown directive type "setting".

    .. setting:: BACKOFF_JITTER

    :setting:`BACKOFF_JITTER` (default: 0.1)

    System Message: ERROR/3 (<stdin>, line 656); backlink

    Unknown interpreted text role "setting".

    Overrides :setting:`RANDOMIZE_DOWNLOAD_DELAY` during backoff.

    System Message: ERROR/3 (<stdin>, line 658); backlink

    Unknown interpreted text role "setting".

  • System Message: ERROR/3 (<stdin>, line 660)

    Unknown directive type "setting".

    .. setting:: BACKOFF_MIN_DELAY

    :setting:`BACKOFF_MIN_DELAY` (default: 1.0)

    System Message: ERROR/3 (<stdin>, line 662); backlink

    Unknown interpreted text role "setting".

    Minimum delay during :ref:`backoff <backoff>`.

    System Message: ERROR/3 (<stdin>, line 664); backlink

    Unknown interpreted text role "ref".

  • System Message: ERROR/3 (<stdin>, line 666)

    Unknown directive type "setting".

    .. setting:: BACKOFF_WINDOW

    :setting:`BACKOFF_WINDOW` (default: 60.0)

    System Message: ERROR/3 (<stdin>, line 668); backlink

    Unknown interpreted text role "setting".

    During :ref:`backoff <backoff>`, after a non-backoff response is received, do not take the next step in backoff reduction until this amount of time has passed and no new backoff feedback has been received.

    System Message: ERROR/3 (<stdin>, line 670); backlink

    Unknown interpreted text role "ref".

    The number of seconds that need to pass since the last non-backoff response without any other for the backoff to move towards the original throttling configuration.

  • System Message: ERROR/3 (<stdin>, line 678)

    Unknown directive type "setting".

    .. setting:: DELAYED_REQUESTS_WARN_THRESHOLD

    :setting:`DELAYED_REQUESTS_WARN_THRESHOLD` (default: 500)

    System Message: ERROR/3 (<stdin>, line 680); backlink

    Unknown interpreted text role "setting".

    While throttled, requests in the :ref:`scheduler <topics-scheduler>` remain in the scheduler.

    System Message: ERROR/3 (<stdin>, line 682); backlink

    Unknown interpreted text role "ref".

    However, requests sent with :meth:`engine.download() <scrapy.core.engine.ExecutionEngine.download>` bypass the scheduler. This includes requests sent by some built-in :ref:`components <topics-components>` and :ref:`inline requests <inline-requests>`.

    System Message: ERROR/3 (<stdin>, line 685); backlink

    Unknown interpreted text role "meth".

    System Message: ERROR/3 (<stdin>, line 685); backlink

    Unknown interpreted text role "ref".

    System Message: ERROR/3 (<stdin>, line 685); backlink

    Unknown interpreted text role "ref".

    When such requests are throttled, they are paused and kept in memory, along with any run time context from the code that is sending them. If they accumulate, they can become a memory issue that may require you to rethink your throttling parameters or crawl strategy.

    :setting:`DELAYED_REQUESTS_WARN_THRESHOLD` defines a threshold for such requests. The first time that this many such requests are being throttled at the same time, a warning is issued.

    System Message: ERROR/3 (<stdin>, line 695); backlink

    Unknown interpreted text role "setting".

  • System Message: ERROR/3 (<stdin>, line 699)

    Unknown directive type "setting".

    .. setting:: RANDOMIZE_DOWNLOAD_DELAY

    :setting:`RANDOMIZE_DOWNLOAD_DELAY` (default: True)

    System Message: ERROR/3 (<stdin>, line 701); backlink

    Unknown interpreted text role "setting".

    Randomize delays by this factor, e.g. if 0.2 randomize delays between delay*0.8 and delay*1.2.

    It can be set to a 2-item list with low and high factors, e.g. [-0.1, 0.3] to randomize delays between delay*0.9 and delay*1.3.

    If True, 0.5 (i.e. ±50%) is used as the randomization factor. If False, no randomization is applied.

API

System Message: ERROR/3 (<stdin>, line 719)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol
    :members:
    :member-order: bysource

System Message: ERROR/3 (<stdin>, line 723)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingManager
    :members: get_response_delay

System Message: ERROR/3 (<stdin>, line 726)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingScopeManagerProtocol
    :members:
    :member-order: bysource

System Message: ERROR/3 (<stdin>, line 730)

Unknown directive type "autoclass".

.. autoclass:: scrapy.throttling.ThrottlingScopeManager

System Message: ERROR/3 (<stdin>, line 732)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.scope_cache

System Message: ERROR/3 (<stdin>, line 733)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.add_scope

System Message: ERROR/3 (<stdin>, line 734)

Unknown directive type "autofunction".

.. autofunction:: scrapy.throttling.update_scope_backoff
</html>