RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

WIP

This commit is contained in:
Adrián Chaves 2025-06-29 02:44:00 +02:00
parent 44b03539df
commit 07fd9265d0
6 changed files with 654 additions and 193 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
docs/topics/components.rst
View File

@ -9,39 +9,24 @@ A Scrapy component is any class whose objects are built using
That includes the classes that you may assign to the following settings:
- :setting:`ADDONS`
- :setting:`DNS_RESOLVER`
- :setting:`DOWNLOAD_HANDLERS`
- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY`
- :setting:`DOWNLOADER_MIDDLEWARES`
- :setting:`DUPEFILTER_CLASS`
- :setting:`EXTENSIONS`
- :setting:`FEED_EXPORTERS`
- :setting:`FEED_STORAGES`
- :setting:`ITEM_PIPELINES`
- :setting:`SCHEDULER`
- :setting:`SCHEDULER_DISK_QUEUE`
- :setting:`SCHEDULER_MEMORY_QUEUE`
- :setting:`SCHEDULER_PRIORITY_QUEUE`
- :setting:`SCHEDULER_START_DISK_QUEUE`
- :setting:`SCHEDULER_START_MEMORY_QUEUE`
- :setting:`SPIDER_MIDDLEWARES`
- :setting:`ADDONS`
- :setting:`DNS_RESOLVER`
- :setting:`DOWNLOAD_HANDLERS`
- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY`
- :setting:`DOWNLOADER_MIDDLEWARES`
- :setting:`DUPEFILTER_CLASS`
- :setting:`EXTENSIONS`
- :setting:`FEED_EXPORTERS`
- :setting:`FEED_STORAGES`
- :setting:`ITEM_PIPELINES`
- :setting:`SCHEDULER`
- :setting:`SCHEDULER_DISK_QUEUE`
- :setting:`SCHEDULER_MEMORY_QUEUE`
- :setting:`SCHEDULER_PRIORITY_QUEUE`
- :setting:`SCHEDULER_START_DISK_QUEUE`
- :setting:`SCHEDULER_START_MEMORY_QUEUE`
- :setting:`SPIDER_MIDDLEWARES`
- :setting:`THROTTLING_MANAGER`
Third-party Scrapy components may also let you define additional Scrapy
components, usually configurable through :ref:`settings <topics-settings>`, to

16
docs/topics/downloader-middleware.rst
View File

@ -886,6 +886,22 @@ Default: ``20``
The maximum number of redirections that will be followed for a single request.
If maximum redirections are exceeded, the request is aborted and ignored.
.. setting:: REDIRECT_MAX_DELAY
REDIRECT_MAX_DELAY
^^^^^^^^^^^^^^^^^^
Default: ``120.0``
If a redirect response provides a ``Retry-After`` header, the redirect is only
followed after the specified delay. This setting caps that delay to a specific
number of seconds. Set to ``0.0`` to disable these delays altogether.
Note that redirect delays only affect the request being redirected, they do not
delay other requests to the same domain. ``Retry-After`` only does that during
:ref:`backoff <retry-after>`.
MetaRefreshMiddleware
---------------------

1
docs/topics/request-response.rst
View File

@ -652,6 +652,7 @@ Those are:
* :reqmeta:`redirect_reasons`
* :reqmeta:`redirect_urls`
* :reqmeta:`referrer_policy`
* :reqmeta:`throttling_scopes`
.. reqmeta:: bindaddress

67
docs/topics/settings.rst
View File

@ -823,49 +823,6 @@ Default: ``True``
Whether to enable downloader stats collection.
.. setting:: DOWNLOAD_DELAY
DOWNLOAD_DELAY
--------------
Default: ``0``
Minimum seconds to wait between 2 consecutive requests to the same domain.
Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting
servers too hard.
Decimal numbers are supported. For example, to send a maximum of 4 requests
every 10 seconds::
DOWNLOAD_DELAY = 2.5
This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY`
setting, which is enabled by default.
When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced
per IP address instead of per domain.
Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain
concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response
time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective
concurrency for that domain is 1. When testing throttling configurations, it
usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first,
and only increase :setting:`DOWNLOAD_DELAY` once
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is
desired.
.. _spider-download_delay-attribute:
.. note::
This delay can be set per spider using :attr:`download_delay` spider attribute.
It is also possible to change this setting per domain, although it requires
non-trivial code. See the implementation of the :ref:`AutoThrottle
<topics-autothrottle>` extension for an example.
.. setting:: DOWNLOAD_HANDLERS
DOWNLOAD_HANDLERS
@ -950,30 +907,6 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2:
.. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption
.. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2
.. setting:: DOWNLOAD_SLOTS
DOWNLOAD_SLOTS
--------------
Default: ``{}``
Allows to define concurrency/delay parameters on per slot (domain) basis:
.. code-block:: python
DOWNLOAD_SLOTS = {
"quotes.toscrape.com": {"concurrency": 1, "delay": 2, "randomize_delay": False},
"books.toscrape.com": {"delay": 3, "randomize_delay": False},
}
.. note::
For other downloader slots default settings values will be used:
- :setting:`DOWNLOAD_DELAY`: ``delay``
- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency``
- :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay``
.. setting:: DOWNLOAD_TIMEOUT

631
docs/topics/throttling.rst
View File

@ -4,32 +4,462 @@
Throttling
==========
Scrapy provides several mechanisms to control the rate at which requests are
sent, to prevent website overloading and handle rate limiting responses.
Sending too many requests too quickly can `overload websites`_. To avoid that,
you must throttle_ your requests. Scrapy can :ref:`throttle requests
<basic-throttling>`, :ref:`handle backoff <backoff>`, and much more.
Basic throttling
================
.. _overload websites: https://en.wikipedia.org/wiki/Denial-of-service_attack
.. _throttle: https://en.wikipedia.org/wiki/Bandwidth_throttling
The main :ref:`settings <topics-settings>` to control throttling are:
.. _basic-throttling:
- :setting:`CONCURRENT_REQUESTS`: The maximum number of total concurrent
requests.
Concurrency and delay
=====================
- :setting:`THROTTLING_BUCKET_CONCURRENCY`: The maximum number of concurrent
requests per :ref:`throttling bucket <throttling-buckets>`.
Use the following :ref:`settings <topics-settings>` to configure the default
throttling for each :ref:`throttling scope <throttling-scopes>`:
- :setting:`THROTTLING_BUCKET_DELAY`: The minimum seconds to wait between
consecutive requests to the same :ref:`throttling bucket
<throttling-buckets>`.
- .. setting:: THROTTLING_CONCURRENCY
**THROTTLING_CONCURRENCY** (default: ``1``)
The maximum number of concurrent requests.
- .. setting:: THROTTLING_DELAY
**THROTTLING_DELAY** (default: ``1.0``)
The minimum seconds to wait between consecutive requests.
- .. setting:: THROTTLING_JITTER
**THROTTLING_JITTER** (default: ``0.5``, i.e. ±50%)
Randomize delays by this factor, i.e. the final delay is a random value
between ``delay*(1-jitter)`` and ``delay*(1+jitter)``.
It can be set to a 2-item list with low and high factors, e.g.
``[-0.1, 0.3]`` to randomize delays between ``delay*0.9`` and
``delay*1.3``.
.. setting:: THROTTLING_SCOPES
Use **THROTTLING_SCOPES** (default: ``{}``) to override these values for
specific throttling scopes:
.. code-block:: python
THROTTLING_SCOPES = {
"books.toscrape.com": {"concurrency": 16, "delay": 0.0},
"example.com": {"jitter": 0.2},
}
:setting:`THROTTLING_SCOPES` can also :ref:`override backoff settings
<scope-backoff>`.
When setting these values, note that:
- :setting:`CONCURRENT_REQUESTS` effectively caps concurrency for any
throttling scope.
- When higher than response time, delay effectively limits concurrency to
``1``.
.. _crawl-delay:
Crawl-Delay directive
---------------------
`Crawl-Delay <https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive>`__
is a non-standard ``robots.txt`` directive that indicates a number of seconds to
wait between requests.
.. setting:: THROTTLING_ROBOTSTXT_OBEY
.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY
If :setting:`ROBOTSTXT_OBEY` and **THROTTLING_ROBOTSTXT_OBEY** are
``True`` (default), valid ``Crawl-Delay`` directives override
:setting:`THROTTLING_CONCURRENCY` and :setting:`THROTTLING_DELAY`. Concurrency
is set to ``1`` and delay is set to the value of ``Crawl-Delay``, capped at
**THROTTLING_ROBOTSTXT_MAX_DELAY** (default: ``60.0``).
If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it
will be respected, but a warning will be logged about the discrepancy with
``Crawl-Delay``. Set ``ignore_robots_txt`` to ``True`` to silence this warning.
.. _backoff:
Backoff
=======
When a response or network error warrants backoff, `exponential backoff`_ is
used to reduce request rate.
.. _exponential backoff: https://en.wikipedia.org/wiki/Exponential_backoff
In such cases, every new request with the same throttling scope is sent with
its delay multiplied by some factor (up to some maximum) or set to some minimum
value (if it was lower), until a request gets a response that does not require
backoff. Once a response that does not require backoff is received, the delay
is gradually reduced back to its original value.
The following settings control backoff behavior:
- .. setting:: BACKOFF_HTTP_CODES
**BACKOFF_HTTP_CODES** (default: ``[429, 502, 503, 504, 520, 521, 522, 523, 524]``)
HTTP response status codes that warrant backoff.
Usually, all codes here should be in :setting:`RETRY_HTTP_CODES` as well,
but not all codes in :setting:`RETRY_HTTP_CODES` need to be here: some bad
responses may require a retry without backoff.
- .. setting:: BACKOFF_EXCEPTIONS
**BACKOFF_EXCEPTIONS**
Default:
.. code-block:: python
[
"twisted.internet.defer.TimeoutError",
"twisted.internet.error.TimeoutError",
"twisted.internet.error.TCPTimedOutError",
"twisted.web.client.ResponseFailed",
]
Exception classes that warrant backoff. Strings are interpreted as import
paths.
Usually, all exceptions here should be in :setting:`RETRY_EXCEPTIONS` as
well, but not all exceptions in :setting:`RETRY_EXCEPTIONS` need to be
here: some errors may require a retry without backoff.
- .. setting:: BACKOFF_FACTOR
**BACKOFF_FACTOR** (default: ``2.0``)
The factor by which the delay is multiplied for each new request sent to a
given throttling scope during backoff.
- .. setting:: BACKOFF_MAX_DELAY
**BACKOFF_MAX_DELAY** (default: ``300.0``)
The maximum delay that can be applied during backoff. If the delay exceeds
this value, it will be capped at this value.
- .. setting:: BACKOFF_MIN_DELAY
**BACKOFF_MIN_DELAY** (default: ``1.0``)
The minimum delay that can be applied during backoff. If the delay is less
than this value, it will be set to this value. Must be higher than ``0.0``.
- .. setting:: BACKOFF_JITTER
**BACKOFF_JITTER** (default: ``0.1``)
Overrides :setting:`THROTTLING_JITTER` during backoff.
When a throttling scope is configured with a **concurrency higher than 1**,
backoff is handled separately per concurrency slot. If at some point all
concurrency slots reach the maximum backoff delay, a “concurrency backoff”
starts, controlled by the following setting:
- .. setting:: BACKOFF_CONCURRENCY_DECREASE_FACTOR
**BACKOFF_CONCURRENCY_DECREASE_FACTOR** (default: ``0.5``)
The factor by which the concurrency is decreased during concurrency
backoff.
.. _scope-backoff:
Backoff settings can be overridden per throttling scope using
:setting:`THROTTLING_SCOPES`:
.. code-block:: python
{
"example.com": {
"backoff": {
"http_codes": [429, 503],
"exceptions": ["builtins.IOError"],
"factor": 1.2,
"max_delay": 180.0,
"min_delay": 5.0,
"jitter": [0.01, 0.33],
"concurrency_decrease_factor": 0.8,
}
},
}
.. _retry-after:
Rate limiting headers
---------------------
The `Retry-After
<https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After>`__
and the `RateLimit-Reset
<https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html#name-ratelimit-reset>`__
HTTP response headers indicate how long to wait before making a follow-up
request.
They are taken into account during :ref:`backoff <backoff>`: their value is
read (the highest if both headers are present), capped at
:setting:`BACKOFF_MAX_DELAY`, and used as a minimum delay, i.e. it is used if
higher than the current delay but ignored if lower.
.. seealso:: :setting:`REDIRECT_MAX_DELAY`
.. _throttling-scopes:
Scopes
======
Throttling scopes represent aspects of requests that can be throttled
independently.
..
For future reference, the “throttling scope” name was taken from
https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html#section-1.4-4.4
.. _default-throttling-scopes:
Default throttling scopes
--------------------------
By default, each request has a single throttling scope representing the domain
or subdomain of the target URL. That is, when you set the concurrency or delay
of a throttling scope, it applies to all requests made to that domain or
subdomain.
For example, https://books.toscrape.com and
https://books.toscrape.com/catalogue/page-2.html both get a
``books.toscrape.com`` throttling scope.
Note however that subdomains are treated as separate throttling scopes by
default. For example, https://toscrape.com gets a ``toscrape.com`` throttling
scope, and ``books.toscrape.com`` and ``toscrape.com`` are considered
unrelated throttling scopes. If you want to change this behavior, see
:ref:`alternative-domain-throttling`.
.. _custom-throttling-scopes:
Customizing throttling scopes
------------------------------
There are 2 ways to customize throttling scopes.
.. reqmeta:: throttling_scopes
For simple use cases, you can use the ``throttling_scopes`` request metadata
key:
.. code-block:: python
Request("https://example.com/", meta={"throttling_scopes": "foo"})
Request("https://example.com/", meta={"throttling_scopes": {"foo", "bar"}})
Request("https://example.com/", meta={"throttling_scopes": {"foo": 1.0, "bar": 2.5}})
.. note:: Throttling scopes set through request metadata remain through the
request lifetime, e.g. throught redirects, even if those change the request
URL.
.. setting:: THROTTLING_MANAGER
For anything else, set **THROTTLING_MANAGER** (default:
:class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component
<topics-components>` that implements the
:class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import
path as a string):
.. code-block:: python
:caption: ``settings.py``
THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager"
.. _throttling-quotas:
Quotas
======
When different requests can consume different amounts of a throttling scope,
you can express this using quotas.
In the :reqmeta:`throttling_scopes` request metadata key and in the
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes` method you use
a :class:`dict` structure where keys are throttling scopes and values are
:class:`float` that indicate the amount of a scope that the request is expected
to consume (it does not need to be exact).
By default, those values are ignored. However, if a call to
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_response_throttling` or
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_exception_throttling`
reports available quotas for one or more throttling scopes, request quotas will
start being tracked and determine which requests can be sent and which cannot.
In fact, everything else being equal,
:class:`~scrapy.pqueues.ScrapyPriorityQueue` prioritizes requests that consume
a higher portion of the available quota, to minimize the risk of those
requests getting stuck.
API
===
.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol
:members:
:member-order: bysource
.. autoclass:: scrapy.throttling.ThrottlingManager
Examples
========
.. _alternative-domain-throttling:
Alternative domain throttling
-----------------------------
If you are not happy with the :ref:`default throttling scope behavior
<default-throttling-scopes>` with regards to domains and subdomains, you can
change it.
Alternative approaches include:
- Using the **highest-level registrable domain** as the throttling scope,
e.g. https://books.toscrape.com and https://toscrape.com both get a
``toscrape.com`` throttling scope.
This allows to apply the same throttling settings to all subdomains of a
registrable domain.
For example:
.. code-block:: python
:caption: ``settings.py``
import tldextract
from scrapy.utils.httpobj import urlparse_cached
class MyThrottlingManager:
def get_request_scopes(self, request):
extracted = tldextract.extract(request.url)
if extracted.domain and extracted.suffix:
return f"{extracted.domain}.{extracted.suffix}"
return urlparse_cached(request).netloc
THROTTLING_MANAGER = MyThrottlingManager
- Using **multiple throttling scopes per request**, one per registrable
domain and for every higher-level subdomain, e.g.
https://books.toscrape.com and https://toscrape.com both get a
``toscrape.com`` throttling scope, but https://books.toscrape.com also
gets a ``books.toscrape.com`` throttling scope.
This allows to apply the same throttling settings to all subdomains of a
registrable domain, but also allows applying further restrictions on each
or on some subdomains.
For example:
.. code-block:: python
:caption: ``settings.py``
import tldextract
from scrapy.utils.httpobj import urlparse_cached
class MyThrottlingManager:
def get_request_scopes(self, request):
extracted = tldextract.extract(request.url)
if not (extracted.domain and extracted.suffix):
return urlparse_cached(request).netloc
scopes = set()
registrable_domain = f"{extracted.domain}.{extracted.suffix}"
scopes.add(registrable_domain)
if extracted.subdomain:
subdomain_parts = extracted.subdomain.split(".")
for i in range(len(subdomain_parts)):
subdomain = ".".join(subdomain_parts[i:])
full_domain = f"{subdomain}.{registrable_domain}"
scopes.add(full_domain)
return scopes
THROTTLING_MANAGER = MyThrottlingManager
THROTTLING_SCOPES = {
"toscrape.com": {"concurrency": 32},
"books.toscrape.com": {"concurrency": 24},
"quotes.toscrape.com": {"concurrency": 16},
}
Here ``books.toscrape.com`` requests can reach 24 concurrency and
``quotes.toscrape.com`` requests can reach 16 concurrency, but never both
at the same time, because that would sum 40 concurrency, and
``toscrape.com`` requests are limited to 32.
Requests can be assigned 1 or more throttling scopes, each with a value.
This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY`
setting, which is enabled by default.
When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced
per IP address instead of per domain.
Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain
concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response
time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective
concurrency for that domain is 1. When testing throttling configurations, it
usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first,
and only increase :setting:`DOWNLOAD_DELAY` once
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is
desired.
.. _spider-download_delay-attribute:
.. note::
This delay can be set per spider using :attr:`download_delay` spider attribute.
It is also possible to change this setting per domain, although it requires
non-trivial code. See the implementation of the :ref:`AutoThrottle
<topics-autothrottle>` extension for an example.
..
TODO: Add a section about the handling of subdomains. By default, Scrapy
should treat subdomains as separate slots, but it should be easy to change
that behavior for specific domains, maybe with some new setting.
.. _throttling-buckets:
Throttling buckets
Throttling scopes
==================
..
@ -40,28 +470,30 @@ Throttling buckets
Settings
========
.. setting:: CONCURRENT_REQUESTS
CONCURRENT_REQUESTS
-------------------
.. setting:: DOWNLOAD_SLOTS
Default: ``16``
DOWNLOAD_SLOTS
--------------
The maximum number of concurrent (i.e. simultaneous) requests that will be
performed by the Scrapy downloader.
Default: ``{}``
.. setting:: CONCURRENT_REQUESTS_PER_DOMAIN
Allows to define concurrency/delay parameters on per slot (domain) basis:
CONCURRENT_REQUESTS_PER_DOMAIN
------------------------------
.. code-block:: python
Default: ``8``
DOWNLOAD_SLOTS = {
"quotes.toscrape.com": {"concurrency": 1, "delay": 2, "randomize_delay": False},
"books.toscrape.com": {"delay": 3, "randomize_delay": False},
}
The maximum number of concurrent (i.e. simultaneous) requests that will be
performed to any single domain.
.. note::
See also: :ref:`topics-autothrottle` and its
:setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` option.
For other downloader slots default settings values will be used:
- :setting:`DOWNLOAD_DELAY`: ``delay``
- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency``
- :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay``
@ -198,12 +630,12 @@ You can configure different throttling settings for different slots using :setti
Advanced Throttling System
===========================
For complex scraping scenarios that require fine-grained control over throttling, Scrapy provides an advanced throttling bucket system that goes beyond simple per-domain limits.
For complex scraping scenarios that require fine-grained control over throttling, Scrapy provides an advanced throttling scope system that goes beyond simple per-domain limits.
Throttling Buckets
------------------
The new throttling system introduces the concept of "throttling buckets" - resources that requests consume and that can become temporarily unavailable when limits are exceeded. Unlike download slots, a single request can require multiple buckets, enabling multi-dimensional throttling.
The new throttling system introduces the concept of "throttling scopes" - resources that requests consume and that can become temporarily unavailable when limits are exceeded. Unlike download slots, a single request can require multiple scopes, enabling multi-dimensional throttling.
Enabling Throttling Buckets
----------------------------
@ -212,80 +644,80 @@ Enabling Throttling Buckets
# settings.py
THROTTLING_ENABLED = True
THROTTLING_BUCKET_MANAGER = "scrapy.throttling.DefaultBucketManager"
THROTTLING_MANAGER = "scrapy.throttling.DefaultBucketManager"
Basic Bucket Usage
------------------
The simplest bucket configuration replicates domain-based throttling:
The simplest scope configuration replicates domain-based throttling:
.. code-block:: python
# Custom bucket manager
# Custom scope manager
class MyBucketManager:
def get_request_buckets(self, request, spider):
def get_request_scopes(self, request, spider):
domain = urlparse(request.url).netloc
return {domain: 1.0} # Consume 1 unit of the domain bucket
return {domain: 1.0} # Consume 1 unit of the domain scope
def process_response(self, response, request, spider):
if response.status == 429: # Too Many Requests
domain = urlparse(request.url).netloc
# Throttle this domain for 60 seconds
self.throttle_bucket(domain, delay=60)
self.throttle_scope(domain, delay=60)
Multi-Dimensional Throttling
-----------------------------
The real power comes from using multiple buckets per request:
The real power comes from using multiple scopes per request:
.. code-block:: python
def get_request_buckets(self, request, spider):
buckets = {}
def get_request_scopes(self, request, spider):
scopes = {}
# Domain-based throttling
domain = urlparse(request.url).netloc
buckets[domain] = 1.0
scopes[domain] = 1.0
# API feature-based throttling
if "browser=true" in request.url:
buckets["browser_rendering"] = 1.0
scopes["browser_rendering"] = 1.0
if "extract=true" in request.url:
buckets["ai_extraction"] = 5.0 # More expensive
scopes["ai_extraction"] = 5.0 # More expensive
# Geographic throttling
if "region=eu" in request.url:
buckets["eu_datacenter"] = 1.0
scopes["eu_datacenter"] = 1.0
return buckets
return scopes
Cost-Based Throttling
---------------------
Some APIs charge different amounts for different operations. The bucket system supports fractional consumption:
Some APIs charge different amounts for different operations. The scope system supports fractional consumption:
.. code-block:: python
def get_request_buckets(self, request, spider):
buckets = {"api_credits": 1.0} # Default cost
def get_request_scopes(self, request, spider):
scopes = {"api_credits": 1.0} # Default cost
if "operation=expensive" in request.url:
buckets["api_credits"] = 10.0 # Costs 10x more
scopes["api_credits"] = 10.0 # Costs 10x more
return buckets
return scopes
def process_response(self, response, request, spider):
# Update actual consumption based on response
if "X-Actual-Cost" in response.headers:
actual_cost = float(response.headers["X-Actual-Cost"])
# Could update bucket consumption here for better accuracy
# Could update scope consumption here for better accuracy
Responding to Server Signals
-----------------------------
The bucket system can respond intelligently to server throttling signals:
The scope system can respond intelligently to server throttling signals:
.. code-block:: python
@ -298,17 +730,17 @@ The bucket system can respond intelligently to server throttling signals:
else:
delay = 60 # Default backoff
# Determine which bucket to throttle based on response
# Determine which scope to throttle based on response
if "rate limit exceeded for API key" in response.text:
self.throttle_bucket("api_key_global", delay=delay)
self.throttle_scope("api_key_global", delay=delay)
elif "too many requests to this endpoint" in response.text:
endpoint = self._extract_endpoint(request.url)
self.throttle_bucket(f"endpoint_{endpoint}", delay=delay)
self.throttle_scope(f"endpoint_{endpoint}", delay=delay)
elif response.status == 503:
# Service unavailable - throttle the entire domain
domain = urlparse(request.url).netloc
self.throttle_bucket(domain, delay=300) # 5 minutes
self.throttle_scope(domain, delay=300) # 5 minutes
Configuration
-------------
@ -319,7 +751,7 @@ The throttling system provides several configuration options:
# settings.py
THROTTLING_ENABLED = True
THROTTLING_BUCKET_MANAGER = "myproject.throttling.CustomBucketManager"
THROTTLING_MANAGER = "myproject.throttling.CustomBucketManager"
# Maximum number of delayed requests to keep in memory
THROTTLING_MAX_DELAYED_REQUESTS = 1000
@ -337,7 +769,7 @@ The throttling system also works with direct downloads made via ``crawler.engine
# In a pipeline or extension
@inlineCallbacks
def process_item(self, item, spider):
# This request will respect throttling buckets
# This request will respect throttling scopes
request = scrapy.Request(item["image_url"])
response = yield spider.crawler.engine.download(request)
# Process response...
@ -376,7 +808,7 @@ Common Patterns
.. code-block:: python
# Use throttling buckets to respect API rate limits
# Use throttling scopes to respect API rate limits
THROTTLING_ENABLED = True
CONCURRENT_REQUESTS_PER_DOMAIN = 5
@ -403,7 +835,7 @@ Scrapy provides several ways to monitor your throttling:
# Check spider.crawler.stats for throttling-related statistics
.. warning::
The AutoThrottle extension is deprecated and not recommended for new projects. It uses a simplistic latency-based approach that doesn't align with modern server throttling patterns. Use the throttling bucket system instead.
The AutoThrottle extension is deprecated and not recommended for new projects. It uses a simplistic latency-based approach that doesn't align with modern server throttling patterns. Use the throttling scope system instead.
Settings Reference
==================
@ -411,31 +843,7 @@ Settings Reference
Global Settings
---------------
.. setting:: CONCURRENT_REQUESTS
**Default**: ``16``
The maximum number of concurrent requests performed by Scrapy.
.. setting:: CONCURRENT_REQUESTS_PER_DOMAIN
**Default**: ``8``
The maximum number of concurrent requests performed to any single domain.
.. setting:: CONCURRENT_REQUESTS_PER_IP
**Default**: ``1``
The maximum number of concurrent requests performed to any single IP address.
.. setting:: DOWNLOAD_DELAY
**Default**: ``0``
The amount of time (in seconds) that the downloader should wait before downloading consecutive pages from the same domain.
.. setting:: RANDOMIZE_DOWNLOAD_DELAY
RANDOMIZE_DOWNLOAD_DELAY
**Default**: ``True``
@ -444,7 +852,7 @@ If enabled, Scrapy will wait a random amount of time (between 0.5 * and 1.5 * ``
Slot Settings
-------------
.. setting:: DOWNLOAD_SLOTS
DOWNLOAD_SLOTS
**Default**: ``{}``
@ -460,13 +868,7 @@ Advanced Throttling Settings
**Default**: ``False``
Enable the advanced throttling bucket system.
.. setting:: THROTTLING_BUCKET_MANAGER
**Default**: ``'scrapy.throttling.DefaultBucketManager'``
A string specifying the throttling bucket manager to use.
Enable the advanced throttling scope system.
.. setting:: THROTTLING_MAX_DELAYED_REQUESTS
@ -479,3 +881,46 @@ Maximum number of delayed requests to keep in memory.
**Default**: ``500``
Warn when the number of delayed requests exceeds this threshold.
..
TODO: Provide real-life examples of throttling configurations, including
exception and error response handling, throttling based on responses,
delay adjustment geared towards rate limits optimization, querying of
external resources for throttling decisions, etc.
- .. setting:: CONCURRENT_REQUESTS
``CONCURRENT_REQUESTS`` (default: ``16``): The maximum number of total
concurrent requests.
..
TODO: Since this setting is more about limiting spider-side resources than
throttling, maybe it does not need to be covered in this page.
..
Implement a signal that can be emitted to change the throttling config of
a specific throttling scope.
..
TODO: Explain how things work, for every setting/parameter, when a request
is assigned multiple scopes.
..
TODO: Explain how scope units work, and give usage examples.
..
TODO: Support backoff in THROTTLING_SCOPES having a cls key with a class
or its import path as a string, and arbitraty kwargs to pass to its
__init__ method, to implement a custom backoff strategy.
..
TODO: Make sure that the API is flexible enough to support scrapy-zyte-api
use cases.

81
scrapy/throttling.py Normal file
View File

@ -0,0 +1,81 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Protocol
from scrapy.utils.httpobj import urlparse_cached
if TYPE_CHECKING:
from collections.abc import Iterable
from scrapy.http import Request, Response
class ThrottlingManagerProtocol(Protocol):
"""A protocol for :setting:`THROTTLING_MANAGER` :ref:`components
<topics-components>`."""
def get_scopes(
self, request: Request
) -> None | str | Iterable[str] | dict[str, float]:
"""Return the :ref:`throttling scopes <throttling-scopes>` that apply
to *request*.
Return ``None`` if no scopes apply, a string for a single scope, an
iterable of strings for multiple scopes, or a dict with scope names as
keys and :ref:`throttling quotas <throttling-quotas>` as values.
"""
def get_response_throttling(
self, response: Response
) -> None | str | Iterable[str] | dict[str, dict[str, Any]]:
"""Return a throttling data update based on *response*.
Return ``None`` if there is nothing new to report, i.e. the response is
not a :ref:`backoff <backoff>` response.
If the response indicates that one or more scopes are currently
exhausted, return a string for a single scope or an iterable of strings
for multiple scopes.
If the response indicates any other information about one or more
scopes, return a dict with scopes as keys and dict values. Dict values
support the following keys:
- ``"delay"``: a float indicating how many seconds to wait before
sending another request for the scope.
- ``"quota"``: a float indicating the remaining :ref:`throttling
quota <throttling-quotas>`.
If ``"quota"`` is not specified, the resource is considered exhausted.
.. code-block:: python
return {
"scope1": {"delay": 5.0},
"scope2": {},
"scope3": {"quota": 42.0},
}
"""
def get_exception_throttling(
self, request: Request, exception: Exception
) -> None | str | Iterable[str] | dict[str, dict[str, Any]]:
"""Return a throttling data update based on *exception* and the
*request* that caused it.
It supports the same return values as :meth:`get_response_throttling`.
"""
class ThrottlingManager:
"""The default :setting:`THROTTLING_MANAGER` class.
It assigns to each request its domain or subdomain as scope and handles
backoff according to :ref:`backoff settings <basic-throttling>`.
"""
def get_scopes(
self, request: Request
) -> None | str | Iterable[str] | dict[str, float]:
return urlparse_cached(request).netloc