diff --git a/docs/topics/components.rst b/docs/topics/components.rst index 56f8c6498..617c668c2 100644 --- a/docs/topics/components.rst +++ b/docs/topics/components.rst @@ -9,39 +9,24 @@ A Scrapy component is any class whose objects are built using That includes the classes that you may assign to the following settings: -- :setting:`ADDONS` - -- :setting:`DNS_RESOLVER` - -- :setting:`DOWNLOAD_HANDLERS` - -- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` - -- :setting:`DOWNLOADER_MIDDLEWARES` - -- :setting:`DUPEFILTER_CLASS` - -- :setting:`EXTENSIONS` - -- :setting:`FEED_EXPORTERS` - -- :setting:`FEED_STORAGES` - -- :setting:`ITEM_PIPELINES` - -- :setting:`SCHEDULER` - -- :setting:`SCHEDULER_DISK_QUEUE` - -- :setting:`SCHEDULER_MEMORY_QUEUE` - -- :setting:`SCHEDULER_PRIORITY_QUEUE` - -- :setting:`SCHEDULER_START_DISK_QUEUE` - -- :setting:`SCHEDULER_START_MEMORY_QUEUE` - -- :setting:`SPIDER_MIDDLEWARES` +- :setting:`ADDONS` +- :setting:`DNS_RESOLVER` +- :setting:`DOWNLOAD_HANDLERS` +- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` +- :setting:`DOWNLOADER_MIDDLEWARES` +- :setting:`DUPEFILTER_CLASS` +- :setting:`EXTENSIONS` +- :setting:`FEED_EXPORTERS` +- :setting:`FEED_STORAGES` +- :setting:`ITEM_PIPELINES` +- :setting:`SCHEDULER` +- :setting:`SCHEDULER_DISK_QUEUE` +- :setting:`SCHEDULER_MEMORY_QUEUE` +- :setting:`SCHEDULER_PRIORITY_QUEUE` +- :setting:`SCHEDULER_START_DISK_QUEUE` +- :setting:`SCHEDULER_START_MEMORY_QUEUE` +- :setting:`SPIDER_MIDDLEWARES` +- :setting:`THROTTLING_MANAGER` Third-party Scrapy components may also let you define additional Scrapy components, usually configurable through :ref:`settings `, to diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 60b6aab78..6bd58e27e 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -886,6 +886,22 @@ Default: ``20`` The maximum number of redirections that will be followed for a single request. If maximum redirections are exceeded, the request is aborted and ignored. +.. setting:: REDIRECT_MAX_DELAY + +REDIRECT_MAX_DELAY +^^^^^^^^^^^^^^^^^^ + +Default: ``120.0`` + +If a redirect response provides a ``Retry-After`` header, the redirect is only +followed after the specified delay. This setting caps that delay to a specific +number of seconds. Set to ``0.0`` to disable these delays altogether. + +Note that redirect delays only affect the request being redirected, they do not +delay other requests to the same domain. ``Retry-After`` only does that during +:ref:`backoff `. + + MetaRefreshMiddleware --------------------- diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 8a907e377..122022bdb 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -652,6 +652,7 @@ Those are: * :reqmeta:`redirect_reasons` * :reqmeta:`redirect_urls` * :reqmeta:`referrer_policy` +* :reqmeta:`throttling_scopes` .. reqmeta:: bindaddress diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 11d1dec5c..3e8de9107 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -823,49 +823,6 @@ Default: ``True`` Whether to enable downloader stats collection. -.. setting:: DOWNLOAD_DELAY - -DOWNLOAD_DELAY --------------- - -Default: ``0`` - -Minimum seconds to wait between 2 consecutive requests to the same domain. - -Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting -servers too hard. - -Decimal numbers are supported. For example, to send a maximum of 4 requests -every 10 seconds:: - - DOWNLOAD_DELAY = 2.5 - -This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY` -setting, which is enabled by default. - -When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced -per IP address instead of per domain. - -Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain -concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response -time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective -concurrency for that domain is 1. When testing throttling configurations, it -usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first, -and only increase :setting:`DOWNLOAD_DELAY` once -:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is -desired. - -.. _spider-download_delay-attribute: - -.. note:: - - This delay can be set per spider using :attr:`download_delay` spider attribute. - -It is also possible to change this setting per domain, although it requires -non-trivial code. See the implementation of the :ref:`AutoThrottle -` extension for an example. - - .. setting:: DOWNLOAD_HANDLERS DOWNLOAD_HANDLERS @@ -950,30 +907,6 @@ The default HTTPS handler uses HTTP/1.1. To use HTTP/2: .. _http2 faq: https://http2.github.io/faq/#does-http2-require-encryption .. _server pushes: https://datatracker.ietf.org/doc/html/rfc7540#section-8.2 -.. setting:: DOWNLOAD_SLOTS - -DOWNLOAD_SLOTS --------------- - -Default: ``{}`` - -Allows to define concurrency/delay parameters on per slot (domain) basis: - - .. code-block:: python - - DOWNLOAD_SLOTS = { - "quotes.toscrape.com": {"concurrency": 1, "delay": 2, "randomize_delay": False}, - "books.toscrape.com": {"delay": 3, "randomize_delay": False}, - } - -.. note:: - - For other downloader slots default settings values will be used: - - - :setting:`DOWNLOAD_DELAY`: ``delay`` - - :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency`` - - :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay`` - .. setting:: DOWNLOAD_TIMEOUT diff --git a/docs/topics/throttling.rst b/docs/topics/throttling.rst index b0d40d213..ce4d5ee00 100644 --- a/docs/topics/throttling.rst +++ b/docs/topics/throttling.rst @@ -4,32 +4,462 @@ Throttling ========== -Scrapy provides several mechanisms to control the rate at which requests are -sent, to prevent website overloading and handle rate limiting responses. +Sending too many requests too quickly can `overload websites`_. To avoid that, +you must throttle_ your requests. Scrapy can :ref:`throttle requests +`, :ref:`handle backoff `, and much more. -Basic throttling -================ +.. _overload websites: https://en.wikipedia.org/wiki/Denial-of-service_attack +.. _throttle: https://en.wikipedia.org/wiki/Bandwidth_throttling -The main :ref:`settings ` to control throttling are: +.. _basic-throttling: -- :setting:`CONCURRENT_REQUESTS`: The maximum number of total concurrent - requests. +Concurrency and delay +===================== -- :setting:`THROTTLING_BUCKET_CONCURRENCY`: The maximum number of concurrent - requests per :ref:`throttling bucket `. +Use the following :ref:`settings ` to configure the default +throttling for each :ref:`throttling scope `: -- :setting:`THROTTLING_BUCKET_DELAY`: The minimum seconds to wait between - consecutive requests to the same :ref:`throttling bucket - `. +- .. setting:: THROTTLING_CONCURRENCY + + **THROTTLING_CONCURRENCY** (default: ``1``) + + The maximum number of concurrent requests. + +- .. setting:: THROTTLING_DELAY + + **THROTTLING_DELAY** (default: ``1.0``) + + The minimum seconds to wait between consecutive requests. + +- .. setting:: THROTTLING_JITTER + + **THROTTLING_JITTER** (default: ``0.5``, i.e. ±50%) + + Randomize delays by this factor, i.e. the final delay is a random value + between ``delay*(1-jitter)`` and ``delay*(1+jitter)``. + + It can be set to a 2-item list with low and high factors, e.g. + ``[-0.1, 0.3]`` to randomize delays between ``delay*0.9`` and + ``delay*1.3``. + +.. setting:: THROTTLING_SCOPES + +Use **THROTTLING_SCOPES** (default: ``{}``) to override these values for +specific throttling scopes: + + .. code-block:: python + + THROTTLING_SCOPES = { + "books.toscrape.com": {"concurrency": 16, "delay": 0.0}, + "example.com": {"jitter": 0.2}, + } + +:setting:`THROTTLING_SCOPES` can also :ref:`override backoff settings +`. + +When setting these values, note that: + +- :setting:`CONCURRENT_REQUESTS` effectively caps concurrency for any + throttling scope. + +- When higher than response time, delay effectively limits concurrency to + ``1``. + + +.. _crawl-delay: + +Crawl-Delay directive +--------------------- + +`Crawl-Delay `__ +is a non-standard ``robots.txt`` directive that indicates a number of seconds to +wait between requests. + +.. setting:: THROTTLING_ROBOTSTXT_OBEY +.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY + +If :setting:`ROBOTSTXT_OBEY` and **THROTTLING_ROBOTSTXT_OBEY** are +``True`` (default), valid ``Crawl-Delay`` directives override +:setting:`THROTTLING_CONCURRENCY` and :setting:`THROTTLING_DELAY`. Concurrency +is set to ``1`` and delay is set to the value of ``Crawl-Delay``, capped at +**THROTTLING_ROBOTSTXT_MAX_DELAY** (default: ``60.0``). + +If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it +will be respected, but a warning will be logged about the discrepancy with +``Crawl-Delay``. Set ``ignore_robots_txt`` to ``True`` to silence this warning. + + +.. _backoff: + +Backoff +======= + +When a response or network error warrants backoff, `exponential backoff`_ is +used to reduce request rate. + +.. _exponential backoff: https://en.wikipedia.org/wiki/Exponential_backoff + +In such cases, every new request with the same throttling scope is sent with +its delay multiplied by some factor (up to some maximum) or set to some minimum +value (if it was lower), until a request gets a response that does not require +backoff. Once a response that does not require backoff is received, the delay +is gradually reduced back to its original value. + +The following settings control backoff behavior: + +- .. setting:: BACKOFF_HTTP_CODES + + **BACKOFF_HTTP_CODES** (default: ``[429, 502, 503, 504, 520, 521, 522, 523, 524]``) + + HTTP response status codes that warrant backoff. + + Usually, all codes here should be in :setting:`RETRY_HTTP_CODES` as well, + but not all codes in :setting:`RETRY_HTTP_CODES` need to be here: some bad + responses may require a retry without backoff. + +- .. setting:: BACKOFF_EXCEPTIONS + + **BACKOFF_EXCEPTIONS** + + Default: + + .. code-block:: python + + [ + "twisted.internet.defer.TimeoutError", + "twisted.internet.error.TimeoutError", + "twisted.internet.error.TCPTimedOutError", + "twisted.web.client.ResponseFailed", + ] + + Exception classes that warrant backoff. Strings are interpreted as import + paths. + + Usually, all exceptions here should be in :setting:`RETRY_EXCEPTIONS` as + well, but not all exceptions in :setting:`RETRY_EXCEPTIONS` need to be + here: some errors may require a retry without backoff. + +- .. setting:: BACKOFF_FACTOR + + **BACKOFF_FACTOR** (default: ``2.0``) + + The factor by which the delay is multiplied for each new request sent to a + given throttling scope during backoff. + +- .. setting:: BACKOFF_MAX_DELAY + + **BACKOFF_MAX_DELAY** (default: ``300.0``) + + The maximum delay that can be applied during backoff. If the delay exceeds + this value, it will be capped at this value. + +- .. setting:: BACKOFF_MIN_DELAY + + **BACKOFF_MIN_DELAY** (default: ``1.0``) + + The minimum delay that can be applied during backoff. If the delay is less + than this value, it will be set to this value. Must be higher than ``0.0``. + +- .. setting:: BACKOFF_JITTER + + **BACKOFF_JITTER** (default: ``0.1``) + + Overrides :setting:`THROTTLING_JITTER` during backoff. + +When a throttling scope is configured with a **concurrency higher than 1**, +backoff is handled separately per concurrency slot. If at some point all +concurrency slots reach the maximum backoff delay, a “concurrency backoff” +starts, controlled by the following setting: + +- .. setting:: BACKOFF_CONCURRENCY_DECREASE_FACTOR + + **BACKOFF_CONCURRENCY_DECREASE_FACTOR** (default: ``0.5``) + + The factor by which the concurrency is decreased during concurrency + backoff. + +.. _scope-backoff: + +Backoff settings can be overridden per throttling scope using +:setting:`THROTTLING_SCOPES`: + +.. code-block:: python + + { + "example.com": { + "backoff": { + "http_codes": [429, 503], + "exceptions": ["builtins.IOError"], + "factor": 1.2, + "max_delay": 180.0, + "min_delay": 5.0, + "jitter": [0.01, 0.33], + "concurrency_decrease_factor": 0.8, + } + }, + } + + +.. _retry-after: + +Rate limiting headers +--------------------- + +The `Retry-After +`__ +and the `RateLimit-Reset +`__ +HTTP response headers indicate how long to wait before making a follow-up +request. + +They are taken into account during :ref:`backoff `: their value is +read (the highest if both headers are present), capped at +:setting:`BACKOFF_MAX_DELAY`, and used as a minimum delay, i.e. it is used if +higher than the current delay but ignored if lower. + +.. seealso:: :setting:`REDIRECT_MAX_DELAY` + + +.. _throttling-scopes: + +Scopes +====== + +Throttling scopes represent aspects of requests that can be throttled +independently. + +.. + For future reference, the “throttling scope” name was taken from + https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html#section-1.4-4.4 + +.. _default-throttling-scopes: + +Default throttling scopes +-------------------------- + +By default, each request has a single throttling scope representing the domain +or subdomain of the target URL. That is, when you set the concurrency or delay +of a throttling scope, it applies to all requests made to that domain or +subdomain. + +For example, https://books.toscrape.com and +https://books.toscrape.com/catalogue/page-2.html both get a +``books.toscrape.com`` throttling scope. + +Note however that subdomains are treated as separate throttling scopes by +default. For example, https://toscrape.com gets a ``toscrape.com`` throttling +scope, and ``books.toscrape.com`` and ``toscrape.com`` are considered +unrelated throttling scopes. If you want to change this behavior, see +:ref:`alternative-domain-throttling`. + + +.. _custom-throttling-scopes: + +Customizing throttling scopes +------------------------------ + +There are 2 ways to customize throttling scopes. + +.. reqmeta:: throttling_scopes + +For simple use cases, you can use the ``throttling_scopes`` request metadata +key: + +.. code-block:: python + + Request("https://example.com/", meta={"throttling_scopes": "foo"}) + Request("https://example.com/", meta={"throttling_scopes": {"foo", "bar"}}) + Request("https://example.com/", meta={"throttling_scopes": {"foo": 1.0, "bar": 2.5}}) + +.. note:: Throttling scopes set through request metadata remain through the + request lifetime, e.g. throught redirects, even if those change the request + URL. + +.. setting:: THROTTLING_MANAGER + +For anything else, set **THROTTLING_MANAGER** (default: +:class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component +` that implements the +:class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import +path as a string): + +.. code-block:: python + :caption: ``settings.py`` + + THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager" + + +.. _throttling-quotas: + +Quotas +====== + +When different requests can consume different amounts of a throttling scope, +you can express this using quotas. + +In the :reqmeta:`throttling_scopes` request metadata key and in the +:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes` method you use +a :class:`dict` structure where keys are throttling scopes and values are +:class:`float` that indicate the amount of a scope that the request is expected +to consume (it does not need to be exact). + +By default, those values are ignored. However, if a call to +:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_response_throttling` or +:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_exception_throttling` +reports available quotas for one or more throttling scopes, request quotas will +start being tracked and determine which requests can be sent and which cannot. + +In fact, everything else being equal, +:class:`~scrapy.pqueues.ScrapyPriorityQueue` prioritizes requests that consume +a higher portion of the available quota, to minimize the risk of those +requests getting stuck. + + +API +=== + +.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol + :members: + :member-order: bysource + +.. autoclass:: scrapy.throttling.ThrottlingManager + + +Examples +======== + +.. _alternative-domain-throttling: + +Alternative domain throttling +----------------------------- + +If you are not happy with the :ref:`default throttling scope behavior +` with regards to domains and subdomains, you can +change it. + +Alternative approaches include: + +- Using the **highest-level registrable domain** as the throttling scope, + e.g. https://books.toscrape.com and https://toscrape.com both get a + ``toscrape.com`` throttling scope. + + This allows to apply the same throttling settings to all subdomains of a + registrable domain. + + For example: + + .. code-block:: python + :caption: ``settings.py`` + + import tldextract + from scrapy.utils.httpobj import urlparse_cached + + + class MyThrottlingManager: + + def get_request_scopes(self, request): + extracted = tldextract.extract(request.url) + if extracted.domain and extracted.suffix: + return f"{extracted.domain}.{extracted.suffix}" + return urlparse_cached(request).netloc + + + THROTTLING_MANAGER = MyThrottlingManager + +- Using **multiple throttling scopes per request**, one per registrable + domain and for every higher-level subdomain, e.g. + https://books.toscrape.com and https://toscrape.com both get a + ``toscrape.com`` throttling scope, but https://books.toscrape.com also + gets a ``books.toscrape.com`` throttling scope. + + This allows to apply the same throttling settings to all subdomains of a + registrable domain, but also allows applying further restrictions on each + or on some subdomains. + + For example: + + .. code-block:: python + :caption: ``settings.py`` + + import tldextract + from scrapy.utils.httpobj import urlparse_cached + + + class MyThrottlingManager: + + def get_request_scopes(self, request): + extracted = tldextract.extract(request.url) + if not (extracted.domain and extracted.suffix): + return urlparse_cached(request).netloc + scopes = set() + registrable_domain = f"{extracted.domain}.{extracted.suffix}" + scopes.add(registrable_domain) + if extracted.subdomain: + subdomain_parts = extracted.subdomain.split(".") + for i in range(len(subdomain_parts)): + subdomain = ".".join(subdomain_parts[i:]) + full_domain = f"{subdomain}.{registrable_domain}" + scopes.add(full_domain) + return scopes + + + THROTTLING_MANAGER = MyThrottlingManager + THROTTLING_SCOPES = { + "toscrape.com": {"concurrency": 32}, + "books.toscrape.com": {"concurrency": 24}, + "quotes.toscrape.com": {"concurrency": 16}, + } + + Here ``books.toscrape.com`` requests can reach 24 concurrency and + ``quotes.toscrape.com`` requests can reach 16 concurrency, but never both + at the same time, because that would sum 40 concurrency, and + ``toscrape.com`` requests are limited to 32. + + + + + + + + + + + +Requests can be assigned 1 or more throttling scopes, each with a value. + + + +This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY` +setting, which is enabled by default. + +When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced +per IP address instead of per domain. + +Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain +concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response +time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective +concurrency for that domain is 1. When testing throttling configurations, it +usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first, +and only increase :setting:`DOWNLOAD_DELAY` once +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is +desired. + +.. _spider-download_delay-attribute: + +.. note:: + + This delay can be set per spider using :attr:`download_delay` spider attribute. + +It is also possible to change this setting per domain, although it requires +non-trivial code. See the implementation of the :ref:`AutoThrottle +` extension for an example. .. TODO: Add a section about the handling of subdomains. By default, Scrapy should treat subdomains as separate slots, but it should be easy to change that behavior for specific domains, maybe with some new setting. -.. _throttling-buckets: - -Throttling buckets +Throttling scopes ================== .. @@ -40,28 +470,30 @@ Throttling buckets Settings ======== -.. setting:: CONCURRENT_REQUESTS -CONCURRENT_REQUESTS -------------------- +.. setting:: DOWNLOAD_SLOTS -Default: ``16`` +DOWNLOAD_SLOTS +-------------- -The maximum number of concurrent (i.e. simultaneous) requests that will be -performed by the Scrapy downloader. +Default: ``{}`` -.. setting:: CONCURRENT_REQUESTS_PER_DOMAIN +Allows to define concurrency/delay parameters on per slot (domain) basis: -CONCURRENT_REQUESTS_PER_DOMAIN ------------------------------- + .. code-block:: python -Default: ``8`` + DOWNLOAD_SLOTS = { + "quotes.toscrape.com": {"concurrency": 1, "delay": 2, "randomize_delay": False}, + "books.toscrape.com": {"delay": 3, "randomize_delay": False}, + } -The maximum number of concurrent (i.e. simultaneous) requests that will be -performed to any single domain. +.. note:: -See also: :ref:`topics-autothrottle` and its -:setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` option. + For other downloader slots default settings values will be used: + + - :setting:`DOWNLOAD_DELAY`: ``delay`` + - :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency`` + - :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay`` @@ -198,12 +630,12 @@ You can configure different throttling settings for different slots using :setti Advanced Throttling System =========================== -For complex scraping scenarios that require fine-grained control over throttling, Scrapy provides an advanced throttling bucket system that goes beyond simple per-domain limits. +For complex scraping scenarios that require fine-grained control over throttling, Scrapy provides an advanced throttling scope system that goes beyond simple per-domain limits. Throttling Buckets ------------------ -The new throttling system introduces the concept of "throttling buckets" - resources that requests consume and that can become temporarily unavailable when limits are exceeded. Unlike download slots, a single request can require multiple buckets, enabling multi-dimensional throttling. +The new throttling system introduces the concept of "throttling scopes" - resources that requests consume and that can become temporarily unavailable when limits are exceeded. Unlike download slots, a single request can require multiple scopes, enabling multi-dimensional throttling. Enabling Throttling Buckets ---------------------------- @@ -212,80 +644,80 @@ Enabling Throttling Buckets # settings.py THROTTLING_ENABLED = True - THROTTLING_BUCKET_MANAGER = "scrapy.throttling.DefaultBucketManager" + THROTTLING_MANAGER = "scrapy.throttling.DefaultBucketManager" Basic Bucket Usage ------------------ -The simplest bucket configuration replicates domain-based throttling: +The simplest scope configuration replicates domain-based throttling: .. code-block:: python - # Custom bucket manager + # Custom scope manager class MyBucketManager: - def get_request_buckets(self, request, spider): + def get_request_scopes(self, request, spider): domain = urlparse(request.url).netloc - return {domain: 1.0} # Consume 1 unit of the domain bucket + return {domain: 1.0} # Consume 1 unit of the domain scope def process_response(self, response, request, spider): if response.status == 429: # Too Many Requests domain = urlparse(request.url).netloc # Throttle this domain for 60 seconds - self.throttle_bucket(domain, delay=60) + self.throttle_scope(domain, delay=60) Multi-Dimensional Throttling ----------------------------- -The real power comes from using multiple buckets per request: +The real power comes from using multiple scopes per request: .. code-block:: python - def get_request_buckets(self, request, spider): - buckets = {} + def get_request_scopes(self, request, spider): + scopes = {} # Domain-based throttling domain = urlparse(request.url).netloc - buckets[domain] = 1.0 + scopes[domain] = 1.0 # API feature-based throttling if "browser=true" in request.url: - buckets["browser_rendering"] = 1.0 + scopes["browser_rendering"] = 1.0 if "extract=true" in request.url: - buckets["ai_extraction"] = 5.0 # More expensive + scopes["ai_extraction"] = 5.0 # More expensive # Geographic throttling if "region=eu" in request.url: - buckets["eu_datacenter"] = 1.0 + scopes["eu_datacenter"] = 1.0 - return buckets + return scopes Cost-Based Throttling --------------------- -Some APIs charge different amounts for different operations. The bucket system supports fractional consumption: +Some APIs charge different amounts for different operations. The scope system supports fractional consumption: .. code-block:: python - def get_request_buckets(self, request, spider): - buckets = {"api_credits": 1.0} # Default cost + def get_request_scopes(self, request, spider): + scopes = {"api_credits": 1.0} # Default cost if "operation=expensive" in request.url: - buckets["api_credits"] = 10.0 # Costs 10x more + scopes["api_credits"] = 10.0 # Costs 10x more - return buckets + return scopes def process_response(self, response, request, spider): # Update actual consumption based on response if "X-Actual-Cost" in response.headers: actual_cost = float(response.headers["X-Actual-Cost"]) - # Could update bucket consumption here for better accuracy + # Could update scope consumption here for better accuracy Responding to Server Signals ----------------------------- -The bucket system can respond intelligently to server throttling signals: +The scope system can respond intelligently to server throttling signals: .. code-block:: python @@ -298,17 +730,17 @@ The bucket system can respond intelligently to server throttling signals: else: delay = 60 # Default backoff - # Determine which bucket to throttle based on response + # Determine which scope to throttle based on response if "rate limit exceeded for API key" in response.text: - self.throttle_bucket("api_key_global", delay=delay) + self.throttle_scope("api_key_global", delay=delay) elif "too many requests to this endpoint" in response.text: endpoint = self._extract_endpoint(request.url) - self.throttle_bucket(f"endpoint_{endpoint}", delay=delay) + self.throttle_scope(f"endpoint_{endpoint}", delay=delay) elif response.status == 503: # Service unavailable - throttle the entire domain domain = urlparse(request.url).netloc - self.throttle_bucket(domain, delay=300) # 5 minutes + self.throttle_scope(domain, delay=300) # 5 minutes Configuration ------------- @@ -319,7 +751,7 @@ The throttling system provides several configuration options: # settings.py THROTTLING_ENABLED = True - THROTTLING_BUCKET_MANAGER = "myproject.throttling.CustomBucketManager" + THROTTLING_MANAGER = "myproject.throttling.CustomBucketManager" # Maximum number of delayed requests to keep in memory THROTTLING_MAX_DELAYED_REQUESTS = 1000 @@ -337,7 +769,7 @@ The throttling system also works with direct downloads made via ``crawler.engine # In a pipeline or extension @inlineCallbacks def process_item(self, item, spider): - # This request will respect throttling buckets + # This request will respect throttling scopes request = scrapy.Request(item["image_url"]) response = yield spider.crawler.engine.download(request) # Process response... @@ -376,7 +808,7 @@ Common Patterns .. code-block:: python - # Use throttling buckets to respect API rate limits + # Use throttling scopes to respect API rate limits THROTTLING_ENABLED = True CONCURRENT_REQUESTS_PER_DOMAIN = 5 @@ -403,7 +835,7 @@ Scrapy provides several ways to monitor your throttling: # Check spider.crawler.stats for throttling-related statistics .. warning:: - The AutoThrottle extension is deprecated and not recommended for new projects. It uses a simplistic latency-based approach that doesn't align with modern server throttling patterns. Use the throttling bucket system instead. + The AutoThrottle extension is deprecated and not recommended for new projects. It uses a simplistic latency-based approach that doesn't align with modern server throttling patterns. Use the throttling scope system instead. Settings Reference ================== @@ -411,31 +843,7 @@ Settings Reference Global Settings --------------- -.. setting:: CONCURRENT_REQUESTS - -**Default**: ``16`` - -The maximum number of concurrent requests performed by Scrapy. - -.. setting:: CONCURRENT_REQUESTS_PER_DOMAIN - -**Default**: ``8`` - -The maximum number of concurrent requests performed to any single domain. - -.. setting:: CONCURRENT_REQUESTS_PER_IP - -**Default**: ``1`` - -The maximum number of concurrent requests performed to any single IP address. - -.. setting:: DOWNLOAD_DELAY - -**Default**: ``0`` - -The amount of time (in seconds) that the downloader should wait before downloading consecutive pages from the same domain. - -.. setting:: RANDOMIZE_DOWNLOAD_DELAY +RANDOMIZE_DOWNLOAD_DELAY **Default**: ``True`` @@ -444,7 +852,7 @@ If enabled, Scrapy will wait a random amount of time (between 0.5 * and 1.5 * `` Slot Settings ------------- -.. setting:: DOWNLOAD_SLOTS +DOWNLOAD_SLOTS **Default**: ``{}`` @@ -460,13 +868,7 @@ Advanced Throttling Settings **Default**: ``False`` -Enable the advanced throttling bucket system. - -.. setting:: THROTTLING_BUCKET_MANAGER - -**Default**: ``'scrapy.throttling.DefaultBucketManager'`` - -A string specifying the throttling bucket manager to use. +Enable the advanced throttling scope system. .. setting:: THROTTLING_MAX_DELAYED_REQUESTS @@ -479,3 +881,46 @@ Maximum number of delayed requests to keep in memory. **Default**: ``500`` Warn when the number of delayed requests exceeds this threshold. + + + +.. + TODO: Provide real-life examples of throttling configurations, including + exception and error response handling, throttling based on responses, + delay adjustment geared towards rate limits optimization, querying of + external resources for throttling decisions, etc. + + + + +- .. setting:: CONCURRENT_REQUESTS + + ``CONCURRENT_REQUESTS`` (default: ``16``): The maximum number of total + concurrent requests. + +.. + TODO: Since this setting is more about limiting spider-side resources than + throttling, maybe it does not need to be covered in this page. + + +.. + Implement a signal that can be emitted to change the throttling config of + a specific throttling scope. + + +.. + TODO: Explain how things work, for every setting/parameter, when a request + is assigned multiple scopes. + +.. + TODO: Explain how scope units work, and give usage examples. + +.. + TODO: Support backoff in THROTTLING_SCOPES having a cls key with a class + or its import path as a string, and arbitraty kwargs to pass to its + __init__ method, to implement a custom backoff strategy. + + +.. + TODO: Make sure that the API is flexible enough to support scrapy-zyte-api + use cases. diff --git a/scrapy/throttling.py b/scrapy/throttling.py new file mode 100644 index 000000000..1d57e1a67 --- /dev/null +++ b/scrapy/throttling.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Protocol + +from scrapy.utils.httpobj import urlparse_cached + +if TYPE_CHECKING: + from collections.abc import Iterable + + from scrapy.http import Request, Response + + +class ThrottlingManagerProtocol(Protocol): + """A protocol for :setting:`THROTTLING_MANAGER` :ref:`components + `.""" + + def get_scopes( + self, request: Request + ) -> None | str | Iterable[str] | dict[str, float]: + """Return the :ref:`throttling scopes ` that apply + to *request*. + + Return ``None`` if no scopes apply, a string for a single scope, an + iterable of strings for multiple scopes, or a dict with scope names as + keys and :ref:`throttling quotas ` as values. + """ + + def get_response_throttling( + self, response: Response + ) -> None | str | Iterable[str] | dict[str, dict[str, Any]]: + """Return a throttling data update based on *response*. + + Return ``None`` if there is nothing new to report, i.e. the response is + not a :ref:`backoff ` response. + + If the response indicates that one or more scopes are currently + exhausted, return a string for a single scope or an iterable of strings + for multiple scopes. + + If the response indicates any other information about one or more + scopes, return a dict with scopes as keys and dict values. Dict values + support the following keys: + + - ``"delay"``: a float indicating how many seconds to wait before + sending another request for the scope. + + - ``"quota"``: a float indicating the remaining :ref:`throttling + quota `. + + If ``"quota"`` is not specified, the resource is considered exhausted. + + .. code-block:: python + + return { + "scope1": {"delay": 5.0}, + "scope2": {}, + "scope3": {"quota": 42.0}, + } + """ + + def get_exception_throttling( + self, request: Request, exception: Exception + ) -> None | str | Iterable[str] | dict[str, dict[str, Any]]: + """Return a throttling data update based on *exception* and the + *request* that caused it. + + It supports the same return values as :meth:`get_response_throttling`. + """ + + +class ThrottlingManager: + """The default :setting:`THROTTLING_MANAGER` class. + + It assigns to each request its domain or subdomain as scope and handles + backoff according to :ref:`backoff settings `. + """ + + def get_scopes( + self, request: Request + ) -> None | str | Iterable[str] | dict[str, float]: + return urlparse_cached(request).netloc