diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 94e27386c..cb9118304 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -141,12 +141,12 @@ parallel. ``quotes.toscrape.com`` is a sandbox meant for scraping practice, so we can safely crawl it faster. Open ``tutorial/settings.py`` and add a -:setting:`THROTTLING_SCOPES` entry that raises the concurrency and lowers the +:setting:`THROTTLER_SCOPES` entry that raises the concurrency and lowers the delay for that domain only: .. code-block:: python - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "quotes.toscrape.com": {"concurrency": 16, "delay": 0.1}, } diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 21096bdfb..ffcff60ae 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -48,7 +48,7 @@ Increase concurrency Concurrency is the number of requests that are processed in parallel. There is a global limit (:setting:`CONCURRENT_REQUESTS`) and an additional per-scope -(per-domain by default) limit (:setting:`THROTTLING_SCOPE_CONCURRENCY`). +(per-domain by default) limit (:setting:`THROTTLER_SCOPE_CONCURRENCY`). The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much diff --git a/docs/topics/components.rst b/docs/topics/components.rst index d7aa5642a..8f9b3865a 100644 --- a/docs/topics/components.rst +++ b/docs/topics/components.rst @@ -24,8 +24,8 @@ That includes the classes that you may assign to the following settings: - :setting:`SCHEDULER_START_DISK_QUEUE` - :setting:`SCHEDULER_START_MEMORY_QUEUE` - :setting:`SPIDER_MIDDLEWARES` -- :setting:`THROTTLING_MANAGER` -- :setting:`THROTTLING_SCOPE_MANAGER` +- :setting:`THROTTLER` +- :setting:`THROTTLER_SCOPE_MANAGER` - :setting:`TWISTED_DNS_RESOLVER` Third-party Scrapy components may also let you define additional Scrapy diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index f4d2d0eaa..a9fd6fd87 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -733,8 +733,8 @@ Those are: * :reqmeta:`redirect_reasons` * :reqmeta:`redirect_urls` * :reqmeta:`referrer_policy` -* :reqmeta:`throttling_delay` -* :reqmeta:`throttling_scopes` +* :reqmeta:`delay` +* :reqmeta:`throttler_scopes` * :reqmeta:`verbatim_url` .. reqmeta:: bindaddress diff --git a/docs/topics/throttling.rst b/docs/topics/throttling.rst index 06799a7f4..34b4f520b 100644 --- a/docs/topics/throttling.rst +++ b/docs/topics/throttling.rst @@ -26,12 +26,12 @@ throttling limits, as do ``toscrape.com`` and ``books.toscrape.com``. The main throttling :ref:`settings ` are: -- .. setting:: THROTTLING_SCOPE_CONCURRENCY +- .. setting:: THROTTLER_SCOPE_CONCURRENCY - :setting:`THROTTLING_SCOPE_CONCURRENCY` (default: ``1``) + :setting:`THROTTLER_SCOPE_CONCURRENCY` (default: ``1``) - Default maximum number of simultaneous requests per :ref:`throttling scope - `. Requests are grouped by domain by default, so this is + Default maximum number of simultaneous requests per :ref:`throttler scope + `. Requests are grouped by domain by default, so this is the maximum number of simultaneous requests per domain. It defines a number of “slots” per scope. Each slot can send 1 request at a @@ -51,29 +51,29 @@ The main throttling :ref:`settings ` are: When configuring these settings, note that: -- :setting:`CONCURRENT_REQUESTS` caps :setting:`THROTTLING_SCOPE_CONCURRENCY`. +- :setting:`CONCURRENT_REQUESTS` caps :setting:`THROTTLER_SCOPE_CONCURRENCY`. - If ``DOWNLOAD_DELAY`` ≥ response time, concurrency is effectively ``1``, because the next request to the domain is not sent until the delay elapses, by which time the previous response has already arrived. -.. [1] You can :ref:`customize ` how requests are grouped +.. [1] You can :ref:`customize ` how requests are grouped for throttling, but domain-based throttling works well in most cases. For more complex domain grouping strategies, see :ref:`alternative-domain-throttling`. -.. setting:: THROTTLING_SCOPES +.. setting:: THROTTLER_SCOPES .. _per-domain-throttling: Per-domain throttling ===================== -The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling +The :setting:`THROTTLER_SCOPES` setting allows you to customize throttling behavior for specific domains [1]_. It is a dict that maps scope names to -:class:`~scrapy.throttling.ThrottlingScopeConfig` dicts. It is empty by default. +:class:`~scrapy.throttler.ThrottlerScopeConfig` dicts. It is empty by default. :command:`startproject` scaffolds a commented-out example entry, so that you can uncomment and edit it to crawl domains you own (or that are meant for scraping) @@ -82,7 +82,7 @@ other domains: .. code-block:: python - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "example.com": {"concurrency": 16, "delay": 0.1}, } @@ -133,7 +133,7 @@ See :ref:`throttling-settings` for :setting:`BACKOFF_EXCEPTIONS`, How backoff works ----------------- -Every :ref:`throttling scope ` keeps a current delay that +Every :ref:`throttler scope ` keeps a current delay that starts at its configured ``"delay"`` (:setting:`DOWNLOAD_DELAY` by default). A **backoff trigger** is a response whose status code is in @@ -181,7 +181,7 @@ Backoff triggers are detected by the Any component can also trigger backoff programmatically for arbitrary scopes — e.g. based on the response body of a specific site — through :meth:`crawler.throttler.back_off() -`. +`. .. _per-scope-backoff: @@ -189,13 +189,13 @@ Per-scope backoff configuration ------------------------------- The global ``BACKOFF_*`` settings can be overridden per scope with the -``"backoff"`` key of a :setting:`THROTTLING_SCOPES` entry, an instance of -:class:`~scrapy.throttling.BackoffConfig`: +``"backoff"`` key of a :setting:`THROTTLER_SCOPES` entry, an instance of +:class:`~scrapy.throttler.BackoffConfig`: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "example.com": { "backoff": { "http_codes": [429, 503], @@ -242,17 +242,17 @@ robots.txt is a non-standard ``robots.txt`` directive that indicates a number of seconds to wait between requests. -.. setting:: THROTTLING_ROBOTSTXT_OBEY -.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY +.. setting:: THROTTLER_ROBOTSTXT_OBEY +.. setting:: THROTTLER_ROBOTSTXT_MAX_DELAY -If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLING_ROBOTSTXT_OBEY` are +If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLER_ROBOTSTXT_OBEY` are ``True`` (default), valid ``Crawl-Delay`` directives override -:setting:`THROTTLING_SCOPE_CONCURRENCY` and :setting:`DOWNLOAD_DELAY`. +:setting:`THROTTLER_SCOPE_CONCURRENCY` and :setting:`DOWNLOAD_DELAY`. Concurrency is set to ``1`` and the delay is raised to at least the ``Crawl-Delay`` value (a larger configured delay is kept), capped at -:setting:`THROTTLING_ROBOTSTXT_MAX_DELAY` (default: ``60.0``). +:setting:`THROTTLER_ROBOTSTXT_MAX_DELAY` (default: ``60.0``). -If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it +If :setting:`THROTTLER_SCOPES` defines a different concurrency or delay, it will be respected, but a warning will be logged about the discrepancy with ``Crawl-Delay``. Set ``ignore_robots_txt`` to ``True`` to silence this warning. @@ -261,9 +261,9 @@ will be respected, but a warning will be logged about the discrepancy with Delaying a scope programmatically ================================= -You can delay a :ref:`throttling scope ` on demand through +You can delay a :ref:`throttler scope ` on demand through :meth:`crawler.throttler.delay_scope() -`: +`: .. skip: next @@ -315,9 +315,9 @@ For example, you might want to throttle API endpoints differently than web pages on the same domain, group requests by content type (images vs HTML), or apply different throttling based on request priority. -.. reqmeta:: throttling_scopes +.. reqmeta:: throttler_scopes -Use the ``throttling_scopes`` request metadata to assign requests to custom +Use the ``throttler_scopes`` request metadata to assign requests to custom throttling groups: .. invisible-code-block: python @@ -326,51 +326,51 @@ throttling groups: .. code-block:: python - Request("https://api.example/", meta={"throttling_scopes": "api"}) + Request("https://api.example/", meta={"throttler_scopes": "api"}) You can also assign multiple throttling groups to a single request: .. code-block:: python - Request("https://api.example/users", meta={"throttling_scopes": {"api", "users"}}) + Request("https://api.example/users", meta={"throttler_scopes": {"api", "users"}}) -You can then use the :setting:`THROTTLING_SCOPES` setting to customize +You can then use the :setting:`THROTTLER_SCOPES` setting to customize throttling for such requests: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "api": {"concurrency": 2}, "users": {"delay": 5.0}, } .. note:: These custom throttling groups persist through redirects. For - redirect-aware throttling assignment, see :ref:`custom-throttling-scopes`. + redirect-aware throttling assignment, see :ref:`custom-throttler-scopes`. -.. reqmeta:: throttling_delay +.. reqmeta:: delay Delaying a single request ------------------------- To hold a single request for a fixed number of seconds before it is sent, -regardless of its scopes, set the ``throttling_delay`` request metadata key: +regardless of its scopes, set the ``delay`` request metadata key: .. code-block:: python - Request("https://example.com/slow", meta={"throttling_delay": 5.0}) + Request("https://example.com/slow", meta={"delay": 5.0}) The delay is applied once, the first time the request reaches the throttling gate. -``throttling_delay`` defines only the *earliest* time the request may be sent, +``delay`` defines only the *earliest* time the request may be sent, not the exact time: once the delay elapses, the request still competes with every other pending request for its scopes. If you want it sent **as soon as** its delay elapses, give it a higher :attr:`~scrapy.Request.priority` too: .. code-block:: python - Request("https://example.com/slow", meta={"throttling_delay": 5.0}, priority=1) + Request("https://example.com/slow", meta={"delay": 5.0}, priority=1) Without a higher priority, a backlog of requests ahead of it in a FIFO queue could keep it waiting well past the configured delay; a higher priority puts it @@ -391,33 +391,33 @@ request normally without letting its outcome trigger :ref:`backoff `: Request("https://example.com/login", meta={"dont_throttle": True}) -.. _throttling-scopes: +.. _throttler-scopes: -Throttling scopes -================= +Throttler scopes +================ -Throttling scopes represent aspects of requests that can be throttled +Throttler scopes represent aspects of requests that can be throttled independently. .. - For future reference, the “throttling scope” name was taken from + For future reference, the “throttler scope” name was taken from https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html#section-1.4-4.4 -.. _custom-throttling-scopes: +.. _custom-throttler-scopes: -Customizing throttling scopes +Customizing throttler scopes ------------------------------ -There are 2 ways to customize throttling scopes. +There are 2 ways to customize throttler scopes. -To **configure existing scopes**, use the :setting:`THROTTLING_SCOPES` setting. +To **configure existing scopes**, use the :setting:`THROTTLER_SCOPES` setting. Its keys are scope names and its values are -:class:`~scrapy.throttling.ThrottlingScopeConfig` dicts, which accept the +:class:`~scrapy.throttler.ThrottlerScopeConfig` dicts, which accept the following keys: ``concurrency`` (:class:`int`) Maximum number of concurrent requests for the scope. Defaults to - :setting:`THROTTLING_SCOPE_CONCURRENCY`. + :setting:`THROTTLER_SCOPE_CONCURRENCY`. ``delay`` (:class:`float`) Minimum seconds between requests for the scope. Defaults to @@ -430,152 +430,152 @@ following keys: a ``[low, high]`` pair multiplies the delay by ``1 + uniform(low, high)``. ``quota`` (:class:`float`) - Maximum :ref:`quota ` consumed per ``window``. + Maximum :ref:`quota ` consumed per ``window``. ``window`` (:class:`float`) - Quota window in seconds. Defaults to :setting:`THROTTLING_WINDOW`. + Quota window in seconds. Defaults to :setting:`THROTTLER_WINDOW`. -``backoff`` (:class:`~scrapy.throttling.BackoffConfig`) +``backoff`` (:class:`~scrapy.throttler.BackoffConfig`) Per-scope :ref:`backoff overrides `. ``manager`` (:class:`str` or :class:`type`) Import path or class of a :ref:`custom scope manager - ` for the scope. + ` for the scope. ``ignore_robots_txt`` (:class:`bool`) Silences the warning logged when this configuration is more aggressive than a :ref:`robots.txt Crawl-delay `. -.. setting:: THROTTLING_MANAGER +.. setting:: THROTTLER To **change how scopes are assigned** (or anything beyond per-scope settings), -set :setting:`THROTTLING_MANAGER` (default: -:class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component +set :setting:`THROTTLER` (default: +:class:`~scrapy.throttler.Throttler`) to a :ref:`component ` that implements the -:class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import +:class:`~scrapy.throttler.ThrottlerProtocol` protocol (or its import path as a string): .. code-block:: python :caption: ``settings.py`` - THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager" + THROTTLER = "myproject.throttling.MyThrottler" -.. _multiple-throttling-scopes: +.. _multiple-throttler-scopes: -Handling of multiple throttling scopes +Handling of multiple throttler scopes -------------------------------------- -When a request has multiple throttling scopes, it is not sent until all of its -throttling scopes allow it. +When a request has multiple throttler scopes, it is not sent until all of its +throttler scopes allow it. -.. _throttling-quotas: +.. _throttler-quotas: -Throttling quotas ------------------ +Throttler quotas +---------------- -When different requests can consume different amounts of a throttling scope, -you can express this using **throttling quotas**. +When different requests can consume different amounts of a throttler scope, +you can express this using **throttler quotas**. -.. setting:: THROTTLING_WINDOW +.. setting:: THROTTLER_WINDOW -Use the :setting:`THROTTLING_WINDOW` setting (default: ``60.0``) or the ``"window"`` -key in the :setting:`THROTTLING_SCOPES` setting to define the time window after -which throttling quotas are reset. +Use the :setting:`THROTTLER_WINDOW` setting (default: ``60.0``) or the ``"window"`` +key in the :setting:`THROTTLER_SCOPES` setting to define the time window after +which throttler quotas are reset. -Then use the :setting:`THROTTLING_SCOPES` setting to define the throttling -quotas for each throttling scope: +Then use the :setting:`THROTTLER_SCOPES` setting to define the throttling +quotas for each throttler scope: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "api.toscrape.com": { "quota": 500.0, }, } -Then, in the :reqmeta:`throttling_scopes` request metadata key or in the return -value of the :meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes` -method, define a :class:`dict` where keys are throttling scopes and values are +Then, in the :reqmeta:`throttler_scopes` request metadata key or in the return +value of the :meth:`~scrapy.throttler.ThrottlerProtocol.get_scopes` +method, define a :class:`dict` where keys are throttler scopes and values are :class:`float` values that indicate the expected quota consumption (it does not need to be exact). -.. _custom-throttling-scope-managers: +.. _custom-throttler-scope-managers: -Customizing throttling scope managers +Customizing throttler scope managers ------------------------------------- -.. setting:: THROTTLING_SCOPE_MANAGER +.. setting:: THROTTLER_SCOPE_MANAGER -The :setting:`THROTTLING_SCOPE_MANAGER` setting (default: -:class:`~scrapy.throttling.ThrottlingScopeManager`) is a :ref:`component +The :setting:`THROTTLER_SCOPE_MANAGER` setting (default: +:class:`~scrapy.throttler.ThrottlerScopeManager`) is a :ref:`component ` that implements the -:class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` (or its import path +:class:`~scrapy.throttler.ThrottlerScopeManagerProtocol` (or its import path as a string): .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPE_MANAGER = "myproject.throttling.MyThrottlingScopeManager" + THROTTLER_SCOPE_MANAGER = "myproject.throttling.MyThrottlerScopeManager" -For each throttling scope, an instance of this class is created to manage any +For each throttler scope, an instance of this class is created to manage any gradual :ref:`backoff ` required at run time. -You can implement your own throttling scope manager if you wish to change the +You can implement your own throttler scope manager if you wish to change the backoff behavior beyond what settings allow. -You can also define a custom throttling scope manager for a specific throttling -scope by setting the ``"manager"`` key in the :setting:`THROTTLING_SCOPES` +You can also define a custom throttler scope manager for a specific throttling +scope by setting the ``"manager"`` key in the :setting:`THROTTLER_SCOPES` setting: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "api.toscrape.com": { - "manager": "myproject.throttling.MyThrottlingScopeManager", + "manager": "myproject.throttling.MyThrottlerScopeManager", }, } Most custom scope managers subclass the default -:class:`~scrapy.throttling.ThrottlingScopeManager` and override only the methods +:class:`~scrapy.throttler.ThrottlerScopeManager` and override only the methods whose behavior they want to change; implementing the -:class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` from scratch is also +:class:`~scrapy.throttler.ThrottlerScopeManagerProtocol` from scratch is also supported. For example, this manager disables exponential :ref:`backoff `, so a scope relies solely on its configured delay and quota: .. code-block:: python :caption: ``myproject/throttling.py`` - from scrapy.throttling import ThrottlingScopeManager + from scrapy.throttler import ThrottlerScopeManager - class FixedWindowScopeManager(ThrottlingScopeManager): + class FixedWindowScopeManager(ThrottlerScopeManager): def record_backoff(self, *args, **kwargs): pass # never back off -.. _throttling-aware-scheduler: +.. _throttler-aware-scheduler: Throttling-aware scheduling =========================== By default, throttling is enforced at the engine, where a request waiting on -its :ref:`throttling scopes ` holds a concurrency slot. In a +its :ref:`throttler scopes ` holds a concurrency slot. In a crawl that mixes heavily-throttled scopes with unthrottled ones, this can let throttled requests starve unthrottled ones that could be sent right away (**head-of-line blocking**; Scrapy logs a warning the first time throttled requests start consuming the global concurrency budget while they wait). -:class:`~scrapy.core.scheduler.ThrottlingAwareScheduler` avoids this. To enable +:class:`~scrapy.core.scheduler.ThrottlerAwareScheduler` avoids this. To enable it: .. code-block:: python :caption: ``settings.py`` - SCHEDULER = "scrapy.core.scheduler.ThrottlingAwareScheduler" - SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.ThrottlingAwarePriorityQueue" + SCHEDULER = "scrapy.core.scheduler.ThrottlerAwareScheduler" + SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.ThrottlerAwarePriorityQueue" -.. autoclass:: scrapy.core.scheduler.ThrottlingAwareScheduler +.. autoclass:: scrapy.core.scheduler.ThrottlerAwareScheduler .. _throttling-examples: @@ -587,14 +587,14 @@ Examples Alternative domain throttling ----------------------------- -If you are not happy with the :ref:`default throttling scope behavior +If you are not happy with the :ref:`default throttler scope behavior ` with regards to domains and subdomains, you can change it. Alternative approaches include: -- Using the **highest-level registrable domain** as the throttling scope, +- Using the **highest-level registrable domain** as the throttler scope, e.g. https://books.toscrape.com and https://toscrape.com both get a - ``toscrape.com`` throttling scope. + ``toscrape.com`` throttler scope. This allows to apply the same throttling settings to all subdomains of a registrable domain. @@ -605,24 +605,24 @@ Alternative approaches include: :caption: ``settings.py`` import tldextract - from scrapy.throttling import ThrottlingManager, scope_cache + from scrapy.throttler import Throttler, scope_cache from scrapy.utils.httpobj import urlparse_cached - class MyThrottlingManager(ThrottlingManager): + class MyThrottler(Throttler): @scope_cache async def get_scopes(self, request): extracted = tldextract.extract(request.url) return extracted.registered_domain or urlparse_cached(request).netloc - THROTTLING_MANAGER = MyThrottlingManager + THROTTLER = MyThrottler -- Using **multiple throttling scopes per request**, one per registrable +- Using **multiple throttler scopes per request**, one per registrable domain and for every higher-level subdomain, e.g. https://books.toscrape.com and https://toscrape.com both get a - ``toscrape.com`` throttling scope, but https://books.toscrape.com also - gets a ``books.toscrape.com`` throttling scope. + ``toscrape.com`` throttler scope, but https://books.toscrape.com also + gets a ``books.toscrape.com`` throttler scope. This allows to apply the same throttling settings to all subdomains of a registrable domain, but also allows applying further restrictions on each @@ -634,11 +634,11 @@ Alternative approaches include: :caption: ``settings.py`` import tldextract - from scrapy.throttling import ThrottlingManager, scope_cache + from scrapy.throttler import Throttler, scope_cache from scrapy.utils.httpobj import urlparse_cached - class MyThrottlingManager(ThrottlingManager): + class MyThrottler(Throttler): @scope_cache async def get_scopes(self, request): extracted = tldextract.extract(request.url) @@ -648,8 +648,8 @@ Alternative approaches include: return {extracted.registered_domain, extracted.fqdn} - THROTTLING_MANAGER = MyThrottlingManager - THROTTLING_SCOPES = { + THROTTLER = MyThrottler + THROTTLER_SCOPES = { "toscrape.com": {"concurrency": 32}, "books.toscrape.com": {"concurrency": 24}, "quotes.toscrape.com": {"concurrency": 16}, @@ -669,16 +669,16 @@ To apply different throttling settings to different endpoints of the same domain and not enforce any common throttling, effectively treating them as different domains: -- Implement a :ref:`throttling manager ` that sets - endpoint-specific throttling scopes for that domain: +- Implement a :ref:`throttler ` that sets + endpoint-specific throttler scopes for that domain: .. code-block:: python - from scrapy.throttling import ThrottlingManager, scope_cache + from scrapy.throttler import Throttler, scope_cache from scrapy.utils.httpobj import urlparse_cached - class MyThrottlingManager(ThrottlingManager): + class MyThrottler(Throttler): @scope_cache async def get_scopes(self, request): parsed_url = urlparse_cached(request) @@ -686,13 +686,13 @@ different domains: return await super().get_scopes(request) return f"{parsed_url.netloc}{parsed_url.path}" -- Use the :setting:`THROTTLING_SCOPES` setting to set different throttling +- Use the :setting:`THROTTLER_SCOPES` setting to set different throttling settings per endpoint: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "api.toscrape.com/fast-endpoint": {"concurrency": 1000, "delay": 0.08}, "api.toscrape.com/slow-endpoint": {"delay": 5.0}, } @@ -707,35 +707,35 @@ Imagine you are sending requests to a web scraping API, e.g. to avoid bans. Unless that API provides a Scrapy plugin to make it easier to use, you may want to: -- Use the :setting:`THROTTLING_SCOPES` setting to increase concurrency for +- Use the :setting:`THROTTLER_SCOPES` setting to increase concurrency for API requests. For example: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "api.toscrape.com": {"concurrency": 1000, "delay": 0.08}, } -- Implement a :ref:`throttling manager ` that: +- Implement a :ref:`throttler ` that: - - Adds a throttling scope for the URL being scraped. + - Adds a throttler scope for the URL being scraped. For example, if you request ``https://api.toscrape.com/?url=https://example.com``, by default it - will get a ``api.toscrape.com`` throttling scope, but it should also - get the ``example.com`` throttling scope: + will get a ``api.toscrape.com`` throttler scope, but it should also + get the ``example.com`` throttler scope: .. code-block:: python from urllib.parse import urlparse - from scrapy.throttling import add_scope, ThrottlingManager, scope_cache + from scrapy.throttler import add_scope, Throttler, scope_cache from scrapy.utils.httpobj import urlparse_cached from w3lib.url import url_query_parameter - class MyThrottlingManager(ThrottlingManager): + class MyThrottler(Throttler): @scope_cache async def get_scopes(self, request): scopes = await super().get_scopes(request) @@ -756,7 +756,7 @@ to: .. code-block:: python - from scrapy.throttling import iter_scopes + from scrapy.throttler import iter_scopes from scrapy.utils.httpobj import urlparse_cached @@ -795,21 +795,21 @@ Cost-capped throttling Imagine you are using an API that charges different requests differently, e.g. based on the features used, and you want to limit how much you spend per time -window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas -` for that: +window (:setting:`THROTTLER_WINDOW`). You can use :ref:`throttler quotas +` for that: -- Implement a :ref:`throttling manager ` that: +- Implement a :ref:`throttler ` that: - - Sets a ``cost`` throttling scope on each request to some estimation + - Sets a ``cost`` throttler scope on each request to some estimation based e.g. on request URL parameters: .. code-block:: python from scrapy.utils.httpobj import urlparse_cached - from scrapy.throttling import ThrottlingManager, scope_cache + from scrapy.throttler import Throttler, scope_cache - class MyThrottlingManager(ThrottlingManager): + class MyThrottler(Throttler): @scope_cache async def get_scopes(self, request): scopes = await super().get_scopes(request) @@ -840,13 +840,13 @@ window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas self.throttler.reconcile_quota("cost", consumed=actual - estimated) return response -- Use the :setting:`THROTTLING_SCOPES` setting to set a maximum cost per time +- Use the :setting:`THROTTLER_SCOPES` setting to set a maximum cost per time window: .. code-block:: python :caption: ``settings.py`` - THROTTLING_SCOPES = { + THROTTLER_SCOPES = { "cost": {"quota": 100.0}, } @@ -858,24 +858,24 @@ window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas Per-IP concurrency limiting --------------------------- -A concurrency limit keyed by IP is just a throttling scope whose id is the +A concurrency limit keyed by IP is just a throttler scope whose id is the request's IP, with a ``concurrency`` limit. A request then carries two scopes, its domain and its IP, and is only sent when **both** allow it (see -:ref:`multiple-throttling-scopes`). +:ref:`multiple-throttler-scopes`). -- Implement a :ref:`throttling manager ` that adds +- Implement a :ref:`throttler ` that adds the request's IP as a second scope: .. code-block:: python import socket - from scrapy.throttling import ThrottlingManager, add_scope, scope_cache + from scrapy.throttler import Throttler, add_scope, scope_cache from scrapy.utils.asyncio import run_in_thread from scrapy.utils.httpobj import urlparse_cached - class IPThrottlingManager(ThrottlingManager): + class IPThrottler(Throttler): @scope_cache async def get_scopes(self, request): scopes = await super().get_scopes(request) @@ -887,6 +887,7 @@ its domain and its IP, and is only sent when **both** allow it (see Additional settings =================== + - .. setting:: BACKOFF_ENABLED :setting:`BACKOFF_ENABLED` (default: ``True``) @@ -933,7 +934,7 @@ Additional settings :setting:`BACKOFF_WINDOW` (default: ``60.0``) Time window, in seconds, used by :ref:`backoff `. A - :ref:`throttling scope ` must go this many seconds + :ref:`throttler scope ` must go this many seconds without a new backoff trigger (an HTTP error code from :setting:`BACKOFF_HTTP_CODES` or an exception from :setting:`BACKOFF_EXCEPTIONS`) before its delay decreases @@ -954,18 +955,18 @@ Additional settings If ``True``, ``0.5`` (i.e. ±50%) is used as the randomization factor. If ``False``, no randomization is applied. -- .. setting:: THROTTLING_DEBUG +- .. setting:: THROTTLER_DEBUG - :setting:`THROTTLING_DEBUG` (default: ``False``) + :setting:`THROTTLER_DEBUG` (default: ``False``) Whether to log :ref:`throttling ` decisions (per-scope delays, backoff steps and recoveries) for debugging. -- .. setting:: THROTTLING_SCOPE_LIMIT +- .. setting:: THROTTLER_SCOPE_LIMIT - :setting:`THROTTLING_SCOPE_LIMIT` (default: ``100000``) + :setting:`THROTTLER_SCOPE_LIMIT` (default: ``100000``) - Maximum number of :ref:`throttling scope ` states kept + Maximum number of :ref:`throttler scope ` states kept in memory at once, to bound memory usage on broad crawls that touch a large number of scopes (e.g. domains). @@ -975,15 +976,15 @@ Additional settings evicted, so the limit may be temporarily exceeded if that many scopes are busy at once. Set to ``0`` to disable the limit. - This complements :setting:`THROTTLING_SCOPE_MAX_IDLE`, which evicts scopes + This complements :setting:`THROTTLER_SCOPE_MAX_IDLE`, which evicts scopes by inactivity time rather than by count. -- .. setting:: THROTTLING_SCOPE_MAX_IDLE +- .. setting:: THROTTLER_SCOPE_MAX_IDLE - :setting:`THROTTLING_SCOPE_MAX_IDLE` (default: ``3600.0``) + :setting:`THROTTLER_SCOPE_MAX_IDLE` (default: ``3600.0``) - Seconds of inactivity after which the state of a :ref:`throttling scope - ` is evicted from memory to bound memory usage on + Seconds of inactivity after which the state of a :ref:`throttler scope + ` is evicted from memory to bound memory usage on long-running crawls. Set to ``0`` to never evict. Scopes in active backoff are never evicted. @@ -992,24 +993,24 @@ Additional settings API === -.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol +.. autoclass:: scrapy.throttler.ThrottlerProtocol :members: :member-order: bysource -.. autoclass:: scrapy.throttling.ThrottlingManager +.. autoclass:: scrapy.throttler.Throttler -.. autoclass:: scrapy.throttling.ThrottlingScopeManagerProtocol +.. autoclass:: scrapy.throttler.ThrottlerScopeManagerProtocol :members: :member-order: bysource -.. autoclass:: scrapy.throttling.ThrottlingScopeManager +.. autoclass:: scrapy.throttler.ThrottlerScopeManager -.. autoclass:: scrapy.pqueues.ThrottlingAwarePriorityQueue +.. autoclass:: scrapy.pqueues.ThrottlerAwarePriorityQueue -.. autoclass:: scrapy.throttling.ThrottlingScopeConfig +.. autoclass:: scrapy.throttler.ThrottlerScopeConfig -.. autoclass:: scrapy.throttling.BackoffConfig +.. autoclass:: scrapy.throttler.BackoffConfig -.. autofunction:: scrapy.throttling.scope_cache -.. autofunction:: scrapy.throttling.add_scope -.. autofunction:: scrapy.throttling.iter_scopes +.. autofunction:: scrapy.throttler.scope_cache +.. autofunction:: scrapy.throttler.add_scope +.. autofunction:: scrapy.throttler.iter_scopes diff --git a/scrapy/core/downloader/__init__.py b/scrapy/core/downloader/__init__.py index 6a3c155d5..ca3162bc1 100644 --- a/scrapy/core/downloader/__init__.py +++ b/scrapy/core/downloader/__init__.py @@ -28,10 +28,7 @@ if TYPE_CHECKING: from scrapy.http import Response from scrapy.settings import BaseSettings from scrapy.signalmanager import SignalManager - from scrapy.throttling import ( - ThrottlingManagerProtocol, - ThrottlingScopeManagerProtocol, - ) + from scrapy.throttler import ThrottlerProtocol, ThrottlerScopeManagerProtocol from scrapy.utils.asyncio import CallLaterResult @@ -92,7 +89,7 @@ class _DeprecatedSlotView: self, downloader: Downloader, key: str, - scope: ThrottlingScopeManagerProtocol, + scope: ThrottlerScopeManagerProtocol, ) -> None: self._downloader = downloader self._key = key @@ -114,10 +111,10 @@ class _DeprecatedSlotView: if r.meta.get(Downloader.DOWNLOAD_SLOT) == self._key } - # This deprecated view reads throttling scope state from private attributes + # This deprecated view reads throttler scope state from private attributes # of the default scope manager rather than through the scope manager # protocol: these are read-only compatibility accessors, so keeping them off - # the protocol avoids forcing custom THROTTLING_SCOPE_MANAGER implementations + # the protocol avoids forcing custom THROTTLER_SCOPE_MANAGER implementations # to provide members that only exist to feed this shim. A custom manager that # lacks the attribute simply falls back to the historical default. @property @@ -168,9 +165,7 @@ class _DeprecatedSlotsView(Mapping[str, _DeprecatedSlotView]): __slots__ = ("_downloader", "_throttler") - def __init__( - self, downloader: Downloader, throttler: ThrottlingManagerProtocol - ) -> None: + def __init__(self, downloader: Downloader, throttler: ThrottlerProtocol) -> None: self._downloader = downloader self._throttler = throttler @@ -216,7 +211,7 @@ class Downloader: ) if self.per_slot_settings: warnings.warn( - "The DOWNLOAD_SLOTS setting is deprecated. Use THROTTLING_SCOPES for " + "The DOWNLOAD_SLOTS setting is deprecated. Use THROTTLER_SCOPES for " "per-domain configuration instead.", category=ScrapyDeprecationWarning, stacklevel=2, @@ -264,7 +259,7 @@ class Downloader: @property def slots(self) -> _DeprecatedSlotsView: warnings.warn( - "Downloader.slots is deprecated. Use the throttling manager API instead.", + "Downloader.slots is deprecated. Use the throttler API instead.", category=ScrapyDeprecationWarning, stacklevel=2, ) diff --git a/scrapy/core/downloader/handlers/_base_http.py b/scrapy/core/downloader/handlers/_base_http.py index 66308389a..b0ddca604 100644 --- a/scrapy/core/downloader/handlers/_base_http.py +++ b/scrapy/core/downloader/handlers/_base_http.py @@ -17,7 +17,7 @@ class BaseHttpDownloadHandler(BaseDownloadHandler, ABC): def _max_per_host_concurrency(settings: BaseSettings) -> int: """Highest per-host concurrency the throttler may admit: the per-domain limit, the default ``other``-scope limit, and any explicit - :setting:`THROTTLING_SCOPES` concurrency. + :setting:`THROTTLER_SCOPES` concurrency. Since :setting:`CONCURRENT_REQUESTS` caps the total number of requests in flight, no host can ever exceed it, so it is also the upper bound of the @@ -26,11 +26,11 @@ class BaseHttpDownloadHandler(BaseDownloadHandler, ABC): global_concurrency = settings.getint("CONCURRENT_REQUESTS") candidates = [ settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN"), - settings.getint("THROTTLING_SCOPE_CONCURRENCY"), + settings.getint("THROTTLER_SCOPE_CONCURRENCY"), ] candidates += [ int(scope["concurrency"]) - for scope in settings.getdict("THROTTLING_SCOPES").values() + for scope in settings.getdict("THROTTLER_SCOPES").values() if "concurrency" in scope ] return min(max(candidates), global_concurrency) diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py index 2dc64c5c3..34acd5051 100644 --- a/scrapy/core/engine.py +++ b/scrapy/core/engine.py @@ -131,9 +131,9 @@ class ExecutionEngine: self._start_request_processing_awaitable: ( asyncio.Future[None] | Deferred[None] | None ) = None - # Requests currently held by the throttling manager, waiting for their + # Requests currently held by the throttler, waiting for their # scopes to allow them through to the downloader. - self._throttling_waiting: set[Request] = set() + self._throttler_waiting: set[Request] = set() # Number of in-flight asynchronous enqueue operations (see # ``_enqueue_request_async``), so the spider is not considered idle # while a request is still on its way into the scheduler. @@ -149,7 +149,7 @@ class ExecutionEngine: # Whether the scheduler exposes enqueue_request_async, cached once per # crawl in ``open_spider_async``; checked for every scheduled request. self._scheduler_enqueues_async: bool = False - self._throttling_backout_warned: bool = False + self._throttler_backout_warned: bool = False downloader_cls: type[Downloader] = load_object(self.settings["DOWNLOADER"]) try: self.scheduler_cls: type[BaseScheduler] = self._get_scheduler_class( @@ -412,40 +412,40 @@ class ExecutionEngine: or bool(self._slot.closing) or self.downloader.needs_backout() or self.scraper.slot.needs_backout() - or self._throttling_needs_backout() + or self._throttler_needs_backout() ) - def _throttling_needs_backout(self) -> bool: - """Apply backpressure while requests are held by the throttling manager. + def _throttler_needs_backout(self) -> bool: + """Apply backpressure while requests are held by the throttler. Requests waiting on throttling do not occupy a downloader concurrency slot, so without this check the engine would keep draining the scheduler into pending throttling waits. Count them together with the in-flight requests against the global concurrency limit. """ - if not self._throttling_waiting: + if not self._throttler_waiting: return False if ( - len(self._throttling_waiting) + len(self.downloader.active) + len(self._throttler_waiting) + len(self.downloader.active) < self.downloader.total_concurrency ): return False - self._maybe_warn_throttling_backout() + self._maybe_warn_throttler_backout() return True - def _maybe_warn_throttling_backout(self) -> None: - if self._throttling_backout_warned: + def _maybe_warn_throttler_backout(self) -> None: + if self._throttler_backout_warned: return - # A throttling-aware scheduler holds time-blocked requests itself instead - # of letting them pile up in _throttling_waiting and back-pressure the + # A throttler-aware scheduler holds time-blocked requests itself instead + # of letting them pile up in _throttler_waiting and back-pressure the # engine here, so the warning does not apply when one is in use. if self._get_next_request_delay is not None: return - self._throttling_backout_warned = True + self._throttler_backout_warned = True logger.warning( "Throttling is holding requests back and they are now consuming the " "global concurrency budget while they wait for a free slot. Consider " - "switching to scrapy.core.scheduler.ThrottlingAwareScheduler, which " + "switching to scrapy.core.scheduler.ThrottlerAwareScheduler, which " "holds throttled requests without consuming concurrency slots.", extra={"spider": self.spider}, ) @@ -523,7 +523,7 @@ class ExecutionEngine: return False if self.downloader.active: # downloader has pending requests return False - if self._throttling_waiting: # requests held by the throttling manager + if self._throttler_waiting: # requests held by the throttler return False if self._scheduling: # requests still on their way into the scheduler return False @@ -612,18 +612,18 @@ class ExecutionEngine: return response_or_request @inlineCallbacks - def _acquire_throttling( + def _acquire_throttler( self, request: Request ) -> Generator[Deferred[Any], Any, None]: """Wait at the throttling gate before *request* is sent, tracking it as held meanwhile.""" - self._throttling_waiting.add(request) + self._throttler_waiting.add(request) throttler = self.crawler.throttler assert throttler is not None try: yield deferred_from_coro(throttler.acquire(request)) finally: - self._throttling_waiting.discard(request) + self._throttler_waiting.discard(request) @inlineCallbacks def _download( @@ -634,12 +634,12 @@ class ExecutionEngine: throttler = self.crawler.throttler assert throttler is not None - # A throttling-aware scheduler reserves the request (a concurrency slot) + # A throttler-aware scheduler reserves the request (a concurrency slot) # before handing it here, so everything past this point runs inside the # try/finally that releases it, even if add_request() were to raise. try: self._slot.add_request(request) - yield self._acquire_throttling(request) + yield self._acquire_throttler(request) result: Response | Request if self._downloader_fetch_needs_spider: result = yield self.downloader.fetch(request, self.spider) diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index fcdb67cab..3ffb6a78c 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, cast from twisted.internet.defer import Deferred # noqa: TC002 from scrapy.spiders import Spider # noqa: TC001 -from scrapy.throttling import iter_scopes +from scrapy.throttler import iter_scopes from scrapy.utils.job import job_dir from scrapy.utils.misc import build_from_crawler, load_object @@ -26,7 +26,7 @@ if TYPE_CHECKING: from scrapy.http.request import Request from scrapy.pqueues import ScrapyPriorityQueue from scrapy.statscollectors import StatsCollector - from scrapy.throttling import ThrottlingManagerProtocol + from scrapy.throttler import ThrottlerProtocol logger = logging.getLogger(__name__) @@ -79,7 +79,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta): Asynchronous counterpart of :meth:`enqueue_request`, following the same return contract. When a scheduler defines it, the engine awaits it *instead of* calling :meth:`enqueue_request`. Define it when enqueuing a - request needs to ``await`` (e.g. to resolve throttling scopes + request needs to ``await`` (e.g. to resolve throttler scopes asynchronously). .. method:: get_next_request_delay() @@ -238,7 +238,7 @@ class Scheduler(BaseScheduler): ------------------------- While pending requests are below the configured values of - :setting:`CONCURRENT_REQUESTS` or :setting:`THROTTLING_SCOPE_CONCURRENCY`, + :setting:`CONCURRENT_REQUESTS` or :setting:`THROTTLER_SCOPE_CONCURRENCY`, those requests are sent concurrently. As a result, the first few requests of a crawl may not follow the desired @@ -410,7 +410,7 @@ class Scheduler(BaseScheduler): Extra positional arguments are forwarded to the underlying queue ``push`` calls; subclasses that store requests under additional keys - (e.g. a throttling scope set) pass them through here. + (e.g. a throttler scope set) pass them through here. """ assert self.stats is not None if self._dqpush(request, *push_args): @@ -535,14 +535,14 @@ class Scheduler(BaseScheduler): json.dump(state, f) -class ThrottlingAwareScheduler(Scheduler): +class ThrottlerAwareScheduler(Scheduler): """A :setting:`SCHEDULER` that only ever hands the engine requests that - their :ref:`throttling scopes ` allow to be sent **right + their :ref:`throttler scopes ` allow to be sent **right now**. The default scheduler hands requests to the engine as concurrency allows and lets them wait at the throttling gate - (:meth:`~scrapy.throttling.ThrottlingManagerProtocol.acquire`) while holding + (:meth:`~scrapy.throttler.ThrottlerProtocol.acquire`) while holding a concurrency slot. When a crawl mixes heavily-throttled scopes with unthrottled ones, enough throttled requests waiting on a clock can fill :setting:`CONCURRENT_REQUESTS` and starve unthrottled requests that could be @@ -551,7 +551,7 @@ class ThrottlingAwareScheduler(Scheduler): (beyond what is needed to track each distinct scope set) and cannot starve other scopes. - It also honors the per-request :reqmeta:`throttling_delay`, holding an + It also honors the per-request :reqmeta:`delay`, holding an individual request back without blocking others that share its scopes, which the default scheduler cannot do. @@ -560,7 +560,7 @@ class ThrottlingAwareScheduler(Scheduler): broken by preferring the least-busy scopes. It requires :setting:`SCHEDULER_PRIORITY_QUEUE` to be set to - :class:`~scrapy.pqueues.ThrottlingAwarePriorityQueue` (or a compatible + :class:`~scrapy.pqueues.ThrottlerAwarePriorityQueue` (or a compatible subclass). """ @@ -569,18 +569,18 @@ class ThrottlingAwareScheduler(Scheduler): if not hasattr(self.mqs, "get_next_request_delay"): raise ValueError( f"{type(self).__name__} requires SCHEDULER_PRIORITY_QUEUE to be " - f"set to a throttling-aware priority queue such as " - f"scrapy.pqueues.ThrottlingAwarePriorityQueue, but the " + f"set to a throttler-aware priority queue such as " + f"scrapy.pqueues.ThrottlerAwarePriorityQueue, but the " f"configured one ({type(self.mqs).__name__}) is not." ) assert self.crawler is not None assert self.crawler.throttler is not None - self._throttler: ThrottlingManagerProtocol = self.crawler.throttler + self._throttler: ThrottlerProtocol = self.crawler.throttler return result def enqueue_request(self, request: Request) -> bool: raise RuntimeError( - "ThrottlingAwareScheduler requires the asynchronous enqueue path; " + "ThrottlerAwareScheduler requires the asynchronous enqueue path; " "enqueue_request_async() is used by the engine instead of " "enqueue_request()." ) diff --git a/scrapy/crawler.py b/scrapy/crawler.py index 0198ef93e..508e6fabf 100644 --- a/scrapy/crawler.py +++ b/scrapy/crawler.py @@ -46,7 +46,7 @@ if TYPE_CHECKING: from scrapy.logformatter import LogFormatter from scrapy.statscollectors import StatsCollector - from scrapy.throttling import ThrottlingManagerProtocol + from scrapy.throttler import ThrottlerProtocol from scrapy.utils.request import RequestFingerprinterProtocol @@ -84,13 +84,13 @@ class Crawler: self.stats: StatsCollector | None = None self.logformatter: LogFormatter | None = None self.request_fingerprinter: RequestFingerprinterProtocol | None = None - self.throttler: ThrottlingManagerProtocol | None = None - """The throttling manager of this crawler, an instance of - :setting:`THROTTLING_MANAGER`. + self.throttler: ThrottlerProtocol | None = None + """The throttler of this crawler, an instance of + :setting:`THROTTLER`. It is ``None`` until the crawl starts. Components can use it to inspect or drive :ref:`throttling ` at run time, e.g. through - :meth:`~scrapy.throttling.ThrottlingManagerProtocol.delay_scope`.""" + :meth:`~scrapy.throttler.ThrottlerProtocol.delay_scope`.""" self.spider: Spider | None = None self.engine: ExecutionEngine | None = None @@ -106,7 +106,7 @@ class Crawler: self.addons.load_settings(self.settings) self._apply_deprecated_spider_attr("download_delay", "DOWNLOAD_DELAY") self._apply_deprecated_spider_attr( - "max_concurrent_requests", "THROTTLING_SCOPE_CONCURRENCY" + "max_concurrent_requests", "THROTTLER_SCOPE_CONCURRENCY" ) self.stats = load_object(self.settings["STATS_CLASS"])(self) @@ -118,7 +118,7 @@ class Crawler: self, ) self.throttler = build_from_crawler( - load_object(self.settings["THROTTLING_MANAGER"]), + load_object(self.settings["THROTTLER"]), self, ) @@ -185,7 +185,7 @@ class Crawler: return warnings.warn( f"The {attr!r} spider attribute is deprecated. Use the {setting} " - f"setting or THROTTLING_SCOPES instead.", + f"setting or THROTTLER_SCOPES instead.", category=ScrapyDeprecationWarning, stacklevel=3, ) diff --git a/scrapy/downloadermiddlewares/backoff.py b/scrapy/downloadermiddlewares/backoff.py index 80c9eb140..37a10ca21 100644 --- a/scrapy/downloadermiddlewares/backoff.py +++ b/scrapy/downloadermiddlewares/backoff.py @@ -4,7 +4,7 @@ import logging from typing import TYPE_CHECKING from scrapy.exceptions import NotConfigured -from scrapy.throttling import iter_scopes +from scrapy.throttler import iter_scopes from scrapy.utils._headers import _parse_ratelimit_reset, _parse_retry_after from scrapy.utils.decorators import _warn_spider_arg from scrapy.utils.misc import _load_objects @@ -16,7 +16,7 @@ if TYPE_CHECKING: import scrapy from scrapy.crawler import Crawler from scrapy.http import Request, Response - from scrapy.throttling import ThrottlingManagerProtocol + from scrapy.throttler import ThrottlerProtocol logger = logging.getLogger(__name__) @@ -28,9 +28,9 @@ class BackoffMiddleware: It observes every response and download exception and, for those matching :setting:`BACKOFF_HTTP_CODES` or :setting:`BACKOFF_EXCEPTIONS` (globally or - per :setting:`THROTTLING_SCOPES` scope), tells the :ref:`throttling manager + per :setting:`THROTTLER_SCOPES` scope), tells the :ref:`throttler ` to back off the request's scopes through its - :meth:`~scrapy.throttling.ThrottlingManagerProtocol.back_off` API. + :meth:`~scrapy.throttler.ThrottlerProtocol.back_off` API. It is enabled by default; set :setting:`BACKOFF_ENABLED` to ``False`` to disable it without removing it from :setting:`DOWNLOADER_MIDDLEWARES`. @@ -46,7 +46,7 @@ class BackoffMiddleware: if not crawler.settings.getbool("BACKOFF_ENABLED"): raise NotConfigured assert crawler.throttler is not None - self._throttler: ThrottlingManagerProtocol = crawler.throttler + self._throttler: ThrottlerProtocol = crawler.throttler settings = crawler.settings self._http_codes: set[int] = { int(code) for code in settings.getlist("BACKOFF_HTTP_CODES") @@ -61,7 +61,7 @@ class BackoffMiddleware: # pre-filter to skip outcomes that no scope backs off on. self._any_http_codes: set[int] = set(self._http_codes) self._any_exceptions: tuple[type[BaseException], ...] = self._exceptions - for scope_id, scope_config in settings.getdict("THROTTLING_SCOPES").items(): + for scope_id, scope_config in settings.getdict("THROTTLER_SCOPES").items(): backoff = scope_config.get("backoff") or {} if "http_codes" in backoff: codes = {int(code) for code in backoff["http_codes"]} diff --git a/scrapy/downloadermiddlewares/redirect.py b/scrapy/downloadermiddlewares/redirect.py index 35b347d08..aa4f2cdab 100644 --- a/scrapy/downloadermiddlewares/redirect.py +++ b/scrapy/downloadermiddlewares/redirect.py @@ -184,7 +184,7 @@ class BaseRedirectMiddleware: """Delay the redirect when *response* carries a ``Retry-After`` header. The delay is capped at :setting:`REDIRECT_MAX_DELAY` and applied through - the :reqmeta:`throttling_delay` request metadata key, which holds back + the :reqmeta:`delay` request metadata key, which holds back this request only (without counting as a :ref:`backoff ` trigger for its scopes). :setting:`REDIRECT_MAX_DELAY` set to ``0`` disables it. @@ -194,11 +194,11 @@ class BaseRedirectMiddleware: retry_after = _parse_retry_after(response) if retry_after is None: return - redirect_request.meta["throttling_delay"] = min(retry_after, self.max_delay) + redirect_request.meta["delay"] = min(retry_after, self.max_delay) # This is a fresh request that may inherit an already-honored delay from # its source request's meta; clear that state so the new delay applies. - redirect_request.meta.pop("_throttling_delayed", None) - redirect_request.meta.pop("_throttling_delay_deadline", None) + redirect_request.meta.pop("_throttler_delayed", None) + redirect_request.meta.pop("_throttler_delay_deadline", None) def _redirect_request_using_get( self, request: Request, response: Response, redirect_url: str diff --git a/scrapy/extensions/throttle.py b/scrapy/extensions/throttle.py index 9f3736c5b..befdfc293 100644 --- a/scrapy/extensions/throttle.py +++ b/scrapy/extensions/throttle.py @@ -14,7 +14,7 @@ if TYPE_CHECKING: from scrapy.crawler import Crawler from scrapy.http import Response - from scrapy.throttling import ThrottlingManagerProtocol + from scrapy.throttler import ThrottlerProtocol logger = logging.getLogger(__name__) @@ -28,7 +28,7 @@ class AutoThrottle: warn( "You have set the AUTOTHROTTLE_ENABLED setting to True, however " - "the AutoThrottle extension is deprecated; use throttling and " + "the AutoThrottle extension is deprecated; use throttler and " "backoff settings instead: " "https://docs.scrapy.org/en/latest/topics/throttling.html", ScrapyDeprecationWarning, @@ -99,9 +99,7 @@ class AutoThrottle: extra={"spider": spider}, ) - def _scope_delay( - self, throttler: ThrottlingManagerProtocol, scope_id: str - ) -> float: + def _scope_delay(self, throttler: ThrottlerProtocol, scope_id: str) -> float: """Return the current delay of *scope_id*, applying AUTOTHROTTLE_START_DELAY the first time the scope is seen.""" delay = throttler.get_scope_delay(scope_id) diff --git a/scrapy/pqueues.py b/scrapy/pqueues.py index 25008cc17..520550874 100644 --- a/scrapy/pqueues.py +++ b/scrapy/pqueues.py @@ -17,7 +17,7 @@ if TYPE_CHECKING: from scrapy import Request from scrapy.crawler import Crawler - from scrapy.throttling import ScopeID, ThrottlingManagerProtocol + from scrapy.throttler import ScopeID, ThrottlerProtocol logger = logging.getLogger(__name__) @@ -329,7 +329,7 @@ class DownloaderAwarePriorityQueue: ) assert crawler.throttler is not None - self._throttler: ThrottlingManagerProtocol = crawler.throttler + self._throttler: ThrottlerProtocol = crawler.throttler self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls self.key: str = key @@ -441,10 +441,10 @@ def _scope_set_from_key(key: str) -> frozenset[ScopeID]: return frozenset(json.loads(key)) -class ThrottlingAwarePriorityQueue: +class ThrottlerAwarePriorityQueue: """Priority queue that only ever pops a request that can be sent right now - based on its :ref:`throttling scope set ` and - per-request :reqmeta:`throttling_delay`. + based on its :ref:`throttler scope set ` and + per-request :reqmeta:`delay`. The downstream queue class must support ``peek``. @@ -460,6 +460,22 @@ class ThrottlingAwarePriorityQueue: persistence directory, and inside it this class creates a subdirectory per scope set, named from a path-safe, order-independent encoding of its scope ids. + + For example, a request whose scope set is ``{"example.com", + "cost:group-1"}`` is stored under a subdirectory derived in two steps: + + #. The scope ids are sorted and JSON-encoded into an order-independent key + (so ``{"example.com", "cost:group-1"}`` and ``{"cost:group-1", + "example.com"}`` map to the same one):: + + ["cost:group-1", "example.com"] + + #. That key is made path-safe: every character outside ``[A-Za-z0-9-._]`` + becomes ``_`` (here the ``:`` and the JSON quotes, brackets and spaces; + the ``-`` and ``.`` are kept), and an MD5 suffix disambiguates keys that + collapse to the same path:: + + __cost_group-1____example.com__-fc6ba2aff8f421bf981b662d77739902 """ @classmethod @@ -491,7 +507,7 @@ class ThrottlingAwarePriorityQueue: ): if slot_startprios and not isinstance(slot_startprios, dict): raise ValueError( - "ThrottlingAwarePriorityQueue accepts ``slot_startprios`` as a " + "ThrottlerAwarePriorityQueue accepts ``slot_startprios`` as a " f"dict; {slot_startprios.__class__!r} instance is passed. Most " "likely, it means the state is created by an incompatible " "priority queue. Only a crawl started with the same priority " @@ -499,7 +515,7 @@ class ThrottlingAwarePriorityQueue: ) assert crawler.throttler is not None - self._throttler: ThrottlingManagerProtocol = crawler.throttler + self._throttler: ThrottlerProtocol = crawler.throttler self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls self.key: str = key @@ -509,15 +525,26 @@ class ThrottlingAwarePriorityQueue: if slot_startprios: for set_key, startprios in slot_startprios.items(): scope_set = _scope_set_from_key(set_key) - self.pqueues[scope_set] = self.pqfactory(scope_set, startprios) + self.pqueues[scope_set] = self._pqfactory(scope_set, startprios) - # Min-heap of (deadline, seq, scope_set, request) for requests held - # back by a per-request throttling_delay; seq keeps ordering stable and - # avoids comparing requests when deadlines tie. + # Requests held back by their own per-request delay wait + # here instead of in their scope-set queue, so a not-yet-due request + # never sits at a queue head and blocks the other requests that share + # its scopes (the head-of-line blocking this scheduler exists to + # avoid). Once the delay elapses, _promote_ready() moves the request + # into its scope-set queue, where it competes normally. + # + # This is a min-heap of (deadline, seq, scope_set, request). heapq + # needs a total order and always compares entries to keep its invariant; + # seq (a monotonic counter) makes the (deadline, seq) prefix totally + # ordered, so a deadline tie is broken there and the scope_set (a + # frozenset, whose < is only the partial subset order) and the request + # (no ordering at all) are never compared. The order among tied + # deadlines is irrelevant beyond being deterministic. self._delayed: list[tuple[float, int, frozenset[ScopeID], Request]] = [] self._delayed_seq: int = 0 - def pqfactory( + def _pqfactory( self, scope_set: frozenset[ScopeID], startprios: Iterable[int] = () ) -> ScrapyPriorityQueue: return ScrapyPriorityQueue( @@ -542,7 +569,7 @@ class ThrottlingAwarePriorityQueue: def _push_to_queue(self, request: Request, scope_set: frozenset[ScopeID]) -> None: if scope_set not in self.pqueues: - self.pqueues[scope_set] = self.pqfactory(scope_set) + self.pqueues[scope_set] = self._pqfactory(scope_set) self.pqueues[scope_set].push(request) def _promote_ready(self, now: float) -> None: @@ -559,7 +586,7 @@ class ThrottlingAwarePriorityQueue: # mark it consumed: the request must not be delayed again, and on resume # it must not re-block its scope set on a stale, no-longer-meaningful # deadline. - request.meta["_throttling_delayed"] = True + request.meta["_throttler_delayed"] = True try: self._push_to_queue(request, scope_set) except ValueError as e: @@ -586,7 +613,7 @@ class ThrottlingAwarePriorityQueue: Among the sendable queues (those whose scope set can be sent right now), the one whose head has the highest request priority is chosen; ties are broken by ascending load (the maximum - :meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scope_load` over the + :meth:`~scrapy.throttler.ThrottlerProtocol.get_scope_load` over the scopes of the queue), i.e. by preferring the least-busy scopes. """ self._promote_ready(time.monotonic()) @@ -618,12 +645,6 @@ class ThrottlingAwarePriorityQueue: del self.pqueues[scope_set] return request - def peek(self) -> Request | None: - selected = self._select() - if selected is None: - return None - return selected[1].peek() - def get_next_request_delay(self) -> float | None: now = time.monotonic() self._promote_ready(now) @@ -639,7 +660,7 @@ class ThrottlingAwarePriorityQueue: continue if delay is None or head_delay < delay: delay = head_delay - # A request held back only by its own throttling_delay is not in any + # A request held back only by its own delay is not in any # scope-set queue, so factor in when the earliest one is due. if self._delayed: next_delayed = max(0.0, self._delayed[0][0] - now) @@ -662,6 +683,3 @@ class ThrottlingAwarePriorityQueue: def __len__(self) -> int: queued = sum(len(x) for x in self.pqueues.values()) if self.pqueues else 0 return queued + len(self._delayed) - - def __contains__(self, scope_set: frozenset[ScopeID]) -> bool: - return scope_set in self.pqueues diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 20859ae30..b4ffa36f3 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -217,16 +217,16 @@ __all__ = [ "TELNETCONSOLE_PORT", "TELNETCONSOLE_USERNAME", "TEMPLATES_DIR", - "THROTTLING_DEBUG", - "THROTTLING_MANAGER", - "THROTTLING_ROBOTSTXT_MAX_DELAY", - "THROTTLING_ROBOTSTXT_OBEY", - "THROTTLING_SCOPES", - "THROTTLING_SCOPE_CONCURRENCY", - "THROTTLING_SCOPE_LIMIT", - "THROTTLING_SCOPE_MANAGER", - "THROTTLING_SCOPE_MAX_IDLE", - "THROTTLING_WINDOW", + "THROTTLER", + "THROTTLER_DEBUG", + "THROTTLER_ROBOTSTXT_MAX_DELAY", + "THROTTLER_ROBOTSTXT_OBEY", + "THROTTLER_SCOPES", + "THROTTLER_SCOPE_CONCURRENCY", + "THROTTLER_SCOPE_LIMIT", + "THROTTLER_SCOPE_MANAGER", + "THROTTLER_SCOPE_MAX_IDLE", + "THROTTLER_WINDOW", "TWISTED_DNS_RESOLVER", "TWISTED_REACTOR", "TWISTED_REACTOR_ENABLED", @@ -607,16 +607,16 @@ TELNETCONSOLE_PASSWORD = None TEMPLATES_DIR = str((Path(__file__).parent / ".." / "templates").resolve()) -THROTTLING_MANAGER = "scrapy.throttling.ThrottlingManager" -THROTTLING_SCOPE_MANAGER = "scrapy.throttling.ThrottlingScopeManager" -THROTTLING_SCOPES = {} -THROTTLING_WINDOW = 60.0 -THROTTLING_ROBOTSTXT_OBEY = True -THROTTLING_ROBOTSTXT_MAX_DELAY = 60.0 -THROTTLING_SCOPE_CONCURRENCY = 1 -THROTTLING_SCOPE_LIMIT = 100000 -THROTTLING_SCOPE_MAX_IDLE = 3600.0 -THROTTLING_DEBUG = False +THROTTLER = "scrapy.throttler.Throttler" +THROTTLER_SCOPE_MANAGER = "scrapy.throttler.ThrottlerScopeManager" +THROTTLER_SCOPES = {} +THROTTLER_WINDOW = 60.0 +THROTTLER_ROBOTSTXT_OBEY = True +THROTTLER_ROBOTSTXT_MAX_DELAY = 60.0 +THROTTLER_SCOPE_CONCURRENCY = 1 +THROTTLER_SCOPE_LIMIT = 100000 +THROTTLER_SCOPE_MAX_IDLE = 3600.0 +THROTTLER_DEBUG = False TWISTED_DNS_RESOLVER = "scrapy.resolver.CachingThreadedResolver" diff --git a/scrapy/templates/project/module/settings.py.tmpl b/scrapy/templates/project/module/settings.py.tmpl index b010a7860..aa3de5312 100644 --- a/scrapy/templates/project/module/settings.py.tmpl +++ b/scrapy/templates/project/module/settings.py.tmpl @@ -21,9 +21,8 @@ ADDONS = {} # Obey robots.txt rules ROBOTSTXT_OBEY = True -# Concurrency and throttling settings #CONCURRENT_REQUESTS = 16 -THROTTLING_SCOPE_CONCURRENCY = 1 +THROTTLER_SCOPE_CONCURRENCY = 1 DOWNLOAD_DELAY = 1 # Disable cookies (enabled by default) diff --git a/scrapy/throttling.py b/scrapy/throttler.py similarity index 88% rename from scrapy/throttling.py rename to scrapy/throttler.py index 6785d607e..171f550ce 100644 --- a/scrapy/throttling.py +++ b/scrapy/throttler.py @@ -33,7 +33,7 @@ logger = logging.getLogger(__name__) class BackoffConfig(TypedDict, total=False): """Per-scope override of the backoff settings. - Used as the value of the ``"backoff"`` key of :class:`ThrottlingScopeConfig` + Used as the value of the ``"backoff"`` key of :class:`ThrottlerScopeConfig` entries. Any key left out falls back to the corresponding global ``BACKOFF_*`` setting. """ @@ -46,8 +46,8 @@ class BackoffConfig(TypedDict, total=False): jitter: float | list[float] -class ThrottlingScopeConfig(TypedDict, total=False): - """Accepted keys of :setting:`THROTTLING_SCOPES` entries. +class ThrottlerScopeConfig(TypedDict, total=False): + """Accepted keys of :setting:`THROTTLER_SCOPES` entries. Every key is optional; missing keys fall back to the matching global setting (e.g. ``delay`` falls back to :setting:`DOWNLOAD_DELAY`). @@ -66,7 +66,7 @@ class ThrottlingScopeConfig(TypedDict, total=False): window: float manager: str | type - """Import path or class of a custom :setting:`THROTTLING_SCOPE_MANAGER` for + """Import path or class of a custom :setting:`THROTTLER_SCOPE_MANAGER` for this scope.""" backoff: BackoffConfig @@ -83,8 +83,8 @@ RequestScopes = None | ScopeID | Iterable[ScopeID] | dict[ScopeID, float | None] def iter_scopes(scopes: RequestScopes) -> Iterable[ScopeID]: """Iterate over the scope IDs of *scopes*, whatever its form. - :class:`~ThrottlingManagerProtocol.get_scopes` (and - :meth:`~ThrottlingManagerProtocol.get_resolved_scopes`) may return a single + :class:`~ThrottlerProtocol.get_scopes` (and + :meth:`~ThrottlerProtocol.get_resolved_scopes`) may return a single scope ID, an iterable of them, a ``{scope_id: quota}`` mapping, or ``None``; this helper normalizes any of those into an iterable of scope IDs, e.g. to react to a request's scopes in a custom middleware. @@ -95,7 +95,7 @@ def iter_scopes(scopes: RequestScopes) -> Iterable[ScopeID]: def iter_scope_values(scopes: RequestScopes) -> Iterable[tuple[ScopeID, float | None]]: """Iterate over *scopes* as ``(scope_id, value)`` pairs. - For dict scopes the value is the expected :ref:`throttling quota + For dict scopes the value is the expected :ref:`throttler quota ` consumption; for every other form the value is ``None``. """ @@ -120,42 +120,42 @@ def _effective_priority(settings: BaseSettings, name: str) -> int: def _default_scope_concurrency(settings: BaseSettings) -> int: - """Return the default concurrency of a throttling scope that does not set + """Return the default concurrency of a throttler scope that does not set its own ``concurrency``. - This is :setting:`THROTTLING_SCOPE_CONCURRENCY`, except that the deprecated + This is :setting:`THROTTLER_SCOPE_CONCURRENCY`, except that the deprecated :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting is bridged in when set at a higher :ref:`priority `. When neither is set explicitly, the historical :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` value is kept for backward compatibility (its default flips to - :setting:`THROTTLING_SCOPE_CONCURRENCY` in a future version; see + :setting:`THROTTLER_SCOPE_CONCURRENCY` in a future version; see :func:`_warn_on_deprecated_concurrency`). """ default_priority = SETTINGS_PRIORITIES["default"] domain_priority = _effective_priority(settings, "CONCURRENT_REQUESTS_PER_DOMAIN") - scope_priority = _effective_priority(settings, "THROTTLING_SCOPE_CONCURRENCY") + scope_priority = _effective_priority(settings, "THROTTLER_SCOPE_CONCURRENCY") if domain_priority > scope_priority: return settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN") if scope_priority > domain_priority: - return settings.getint("THROTTLING_SCOPE_CONCURRENCY") + return settings.getint("THROTTLER_SCOPE_CONCURRENCY") # Equal priority: on an explicit (higher-than-default) tie the new setting # wins; when neither is set (both at "default") keep the historical # per-domain value so existing behavior is preserved. if domain_priority <= default_priority: return settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN") - return settings.getint("THROTTLING_SCOPE_CONCURRENCY") + return settings.getint("THROTTLER_SCOPE_CONCURRENCY") def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None: """Warn about the concurrency settings bridged by :func:`_default_scope_concurrency`. Call once per crawl (see - :meth:`ThrottlingManager.__init__`). + :meth:`Throttler.__init__`). When :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is set explicitly, warn that it is deprecated. When neither concurrency setting is set explicitly, warn that the effective per-scope concurrency is still the deprecated setting's value (kept for backward compatibility) and will drop to - :setting:`THROTTLING_SCOPE_CONCURRENCY`'s default once the deprecated + :setting:`THROTTLER_SCOPE_CONCURRENCY`'s default once the deprecated setting is removed, so users can pin it explicitly.""" default_priority = SETTINGS_PRIORITIES["default"] domain_set = ( @@ -163,12 +163,12 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None: > default_priority ) scope_set = ( - _effective_priority(settings, "THROTTLING_SCOPE_CONCURRENCY") > default_priority + _effective_priority(settings, "THROTTLER_SCOPE_CONCURRENCY") > default_priority ) if domain_set: warnings.warn( "The CONCURRENT_REQUESTS_PER_DOMAIN setting is deprecated, use " - "THROTTLING_SCOPE_CONCURRENCY instead.", + "THROTTLER_SCOPE_CONCURRENCY instead.", category=ScrapyDeprecationWarning, stacklevel=2, ) @@ -178,14 +178,14 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None: # guarded by test_deprecated_concurrency_defaults_differ rather than at # run time, so a crawl is never aborted over it. current = settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN") - future = settings.getint("THROTTLING_SCOPE_CONCURRENCY") + future = settings.getint("THROTTLER_SCOPE_CONCURRENCY") warnings.warn( f"The effective per-scope (per-domain) concurrency is {current}, " f"the default of the deprecated CONCURRENT_REQUESTS_PER_DOMAIN " f"setting, which is still respected for backward compatibility. " f"Once CONCURRENT_REQUESTS_PER_DOMAIN is removed, it will drop to " - f"{future}, the default of THROTTLING_SCOPE_CONCURRENCY. Set " - f"THROTTLING_SCOPE_CONCURRENCY explicitly to choose a value and " + f"{future}, the default of THROTTLER_SCOPE_CONCURRENCY. Set " + f"THROTTLER_SCOPE_CONCURRENCY explicitly to choose a value and " f"silence this warning.", category=ScrapyDeprecationWarning, stacklevel=2, @@ -195,7 +195,7 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None: def _warn_on_unachievable_concurrency(settings: BaseSettings) -> None: """Warn about configured concurrency limits that exceed :setting:`CONCURRENT_REQUESTS`. Call once per crawl (see - :meth:`ThrottlingManager.__init__`). + :meth:`Throttler.__init__`). :setting:`CONCURRENT_REQUESTS` caps the total number of requests in flight, so a per-scope (or per-domain) concurrency limit above it can never be @@ -204,12 +204,12 @@ def _warn_on_unachievable_concurrency(settings: BaseSettings) -> None: global_concurrency = settings.getint("CONCURRENT_REQUESTS") offenders: list[str] = [ f"{name}={settings.getint(name)}" - for name in ("CONCURRENT_REQUESTS_PER_DOMAIN", "THROTTLING_SCOPE_CONCURRENCY") + for name in ("CONCURRENT_REQUESTS_PER_DOMAIN", "THROTTLER_SCOPE_CONCURRENCY") if settings.getint(name) > global_concurrency ] offenders += [ - f"THROTTLING_SCOPES[{scope_id!r}]['concurrency']={config['concurrency']}" - for scope_id, config in settings.getdict("THROTTLING_SCOPES").items() + f"THROTTLER_SCOPES[{scope_id!r}]['concurrency']={config['concurrency']}" + for scope_id, config in settings.getdict("THROTTLER_SCOPES").items() if config.get("concurrency") is not None and int(config["concurrency"]) > global_concurrency ] @@ -249,8 +249,8 @@ def add_scope( dict. This is a utility function to help extending the output of - :meth:`~ThrottlingManagerProtocol.get_scopes`, e.g. in - :class:`ThrottlingManager` subclasses. + :meth:`~ThrottlerProtocol.get_scopes`, e.g. in + :class:`Throttler` subclasses. Adding a scope with a *value* fails if it is already present, so an existing :ref:`quota ` is never silently overwritten; adding it @@ -266,21 +266,21 @@ def add_scope( return result -class ThrottlingManagerProtocol(Protocol): - """A protocol for :setting:`THROTTLING_MANAGER` :ref:`components +class ThrottlerProtocol(Protocol): + """A protocol for :setting:`THROTTLER` :ref:`components `.""" async def get_scopes(self, request: Request) -> RequestScopes: - """Return the :ref:`throttling scopes ` that apply + """Return the :ref:`throttler scopes ` that apply to *request*. Return ``None`` if no scopes apply, a string for a single scope, an iterable of strings for multiple scopes, or a dict with scope names as - keys and :ref:`throttling quotas ` as values. + keys and :ref:`throttler quotas ` as values. """ def get_resolved_scopes(self, request: Request) -> RequestScopes: - """Return the :ref:`throttling scopes ` under which + """Return the :ref:`throttler scopes ` under which *request* was (or will be) sent, without re-resolving them. This is the synchronous counterpart of :meth:`get_scopes`: it returns @@ -314,8 +314,8 @@ class ThrottlingManagerProtocol(Protocol): *and* a concurrency slot is free in every scope. This is the synchronous, non-blocking counterpart of :meth:`acquire`, - used by a :ref:`throttling-aware scheduler - ` to decide whether a request can be + used by a :ref:`throttler-aware scheduler + ` to decide whether a request can be dequeued now. It assumes the scopes of *request* have already been resolved (e.g. by an earlier :meth:`get_scopes` call at enqueue time). """ @@ -325,7 +325,7 @@ class ThrottlingManagerProtocol(Protocol): scopes and mark *request* as reserved, so that a later :meth:`acquire` for it returns immediately without reserving again. - A :ref:`throttling-aware scheduler ` calls + A :ref:`throttler-aware scheduler ` calls this when it decides to dequeue *request* (after :meth:`is_ready` returned ``True``). The reservation is released by :meth:`release`. """ @@ -335,8 +335,8 @@ class ThrottlingManagerProtocol(Protocol): *request* would be open, or ``None`` if no time-based gate is currently blocking it (only a concurrency slot could be). - Used by a :ref:`throttling-aware scheduler - ` to schedule a wakeup when all pending + Used by a :ref:`throttler-aware scheduler + ` to schedule a wakeup when all pending requests are time-blocked. """ @@ -354,19 +354,19 @@ class ThrottlingManagerProtocol(Protocol): active sends divided by its concurrency limit (or by the global :setting:`CONCURRENT_REQUESTS` when the scope has no explicit limit). - Used by a :ref:`throttling-aware scheduler - ` to balance dequeuing across scopes, + Used by a :ref:`throttler-aware scheduler + ` to balance dequeuing across scopes, preferring the least-loaded ones. """ def get_request_delay(self, request: Request, now: float | None = None) -> float: """Return how many seconds *request* must still be held individually - because of its :reqmeta:`throttling_delay`, or ``0.0`` if it has none + because of its :reqmeta:`delay`, or ``0.0`` if it has none or it has already elapsed. The one-time delay is started on the first call. Unlike a scope delay, this affects only *request*: a - :ref:`throttling-aware scheduler ` must + :ref:`throttler-aware scheduler ` must hold the request back on its own, **without** blocking other requests that share its scopes. """ @@ -423,7 +423,7 @@ class ThrottlingManagerProtocol(Protocol): consumed: float | None = None, remaining: float | None = None, ) -> None: - """Reconcile the :ref:`throttling quota ` of each of + """Reconcile the :ref:`throttler quota ` of each of *scopes* with an actually *consumed* amount (a delta to add) or a *remaining* amount (an absolute value), correcting the estimate used when requests were sent. @@ -444,8 +444,8 @@ class ThrottlingManagerProtocol(Protocol): directly (e.g. an adaptive-delay extension). """ - def get_scope_manager(self, scope_id: str) -> ThrottlingScopeManagerProtocol: - """Return the :class:`ThrottlingScopeManagerProtocol` instance handling + def get_scope_manager(self, scope_id: str) -> ThrottlerScopeManagerProtocol: + """Return the :class:`ThrottlerScopeManagerProtocol` instance handling the scope identified by *scope_id*, creating it if necessary.""" @@ -456,16 +456,16 @@ _GetScopesMethod = TypeVar( # Request.meta key under which scope_cache persists the resolved scopes so that # they survive a request being serialized to and restored from a disk queue. -_RESOLVED_SCOPES_META_KEY = "_throttling_resolved_scopes" +_RESOLVED_SCOPES_META_KEY = "_throttler_resolved_scopes" def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod: - """Decorator for :meth:`~ThrottlingManagerProtocol.get_scopes` + """Decorator for :meth:`~ThrottlerProtocol.get_scopes` implementations that persists the resolved scopes on ``request.meta``. The readers of the resolved scopes — the synchronous readiness API of a - :ref:`throttling-aware scheduler ` and - :meth:`~ThrottlingManagerProtocol.get_resolved_scopes` — read this persisted + :ref:`throttler-aware scheduler ` and + :meth:`~ThrottlerProtocol.get_resolved_scopes` — read this persisted value instead of resolving the scopes again, so they stay cheap and consistent, and it survives a request being serialized to and restored from a :ref:`disk queue ` (which is @@ -483,10 +483,10 @@ def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod: .. code-block:: python from scrapy.utils.httpobj import urlparse_cached - from scrapy.throttling import scope_cache + from scrapy.throttler import scope_cache - class MyThrottlingManager: + class MyThrottler: @scope_cache async def get_scopes(self, request): return urlparse_cached(request).hostname or "" @@ -505,8 +505,8 @@ def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod: return wrapper # type: ignore[return-value] -class ThrottlingManager: - """The default :setting:`THROTTLING_MANAGER` class. +class Throttler: + """The default :setting:`THROTTLER` class. It assigns to each request its domain or subdomain as scope and handles backoff according to :ref:`backoff settings `. @@ -520,13 +520,13 @@ class ThrottlingManager: self.crawler = crawler _warn_on_deprecated_concurrency(crawler.settings) _warn_on_unachievable_concurrency(crawler.settings) - self._debug = crawler.settings.getbool("THROTTLING_DEBUG") - self._max_idle = crawler.settings.getfloat("THROTTLING_SCOPE_MAX_IDLE") + self._debug = crawler.settings.getbool("THROTTLER_DEBUG") + self._max_idle = crawler.settings.getfloat("THROTTLER_SCOPE_MAX_IDLE") self._robotstxt_obey = crawler.settings.getbool( "ROBOTSTXT_OBEY" - ) and crawler.settings.getbool("THROTTLING_ROBOTSTXT_OBEY") + ) and crawler.settings.getbool("THROTTLER_ROBOTSTXT_OBEY") self._robotstxt_max_delay = crawler.settings.getfloat( - "THROTTLING_ROBOTSTXT_MAX_DELAY" + "THROTTLER_ROBOTSTXT_MAX_DELAY" ) self._default_useragent: str = crawler.settings["USER_AGENT"] self._robotstxt_useragent: str | None = crawler.settings["ROBOTSTXT_USER_AGENT"] @@ -535,40 +535,40 @@ class ThrottlingManager: self._on_robots_parsed, signal=signals.robots_parsed ) self._default_scope_manager_cls = load_object( - crawler.settings["THROTTLING_SCOPE_MANAGER"] + crawler.settings["THROTTLER_SCOPE_MANAGER"] ) self._scopes_config: dict[str, dict[str, Any]] = self._merge_download_slots( crawler.settings ) # Ordered by least-recently-used first (see get_scope_manager), so the - # scope limit can evict the coldest idle scopes (see THROTTLING_SCOPE_LIMIT). - self._scope_managers: OrderedDict[ScopeID, ThrottlingScopeManagerProtocol] = ( + # scope limit can evict the coldest idle scopes (see THROTTLER_SCOPE_LIMIT). + self._scope_managers: OrderedDict[ScopeID, ThrottlerScopeManagerProtocol] = ( OrderedDict() ) - self._scope_limit: int = crawler.settings.getint("THROTTLING_SCOPE_LIMIT") + self._scope_limit: int = crawler.settings.getint("THROTTLER_SCOPE_LIMIT") self._last_eviction: float | None = None # Concurrency slots reserved by acquire(), to be released once the # request finishes downloading. self._reserved: WeakKeyDictionary[ - Request, list[tuple[ThrottlingScopeManagerProtocol, float | None]] + Request, list[tuple[ThrottlerScopeManagerProtocol, float | None]] ] = WeakKeyDictionary() @staticmethod def _merge_download_slots(settings: BaseSettings) -> dict[str, dict[str, Any]]: """Return the effective per-scope configuration, merging the deprecated - :setting:`DOWNLOAD_SLOTS` setting into :setting:`THROTTLING_SCOPES`. + :setting:`DOWNLOAD_SLOTS` setting into :setting:`THROTTLER_SCOPES`. - Each ``DOWNLOAD_SLOTS`` entry is translated to a throttling scope keyed + Each ``DOWNLOAD_SLOTS`` entry is translated to a throttler scope keyed by the same slot name (the default manager keys domain scopes by host name, which is what download slots used too): ``concurrency`` and ``delay`` map directly, and the ``randomize_delay`` boolean maps to a ``jitter`` magnitude (the historical ±50%, or none). An explicit - ``THROTTLING_SCOPES`` entry for the same scope takes precedence over the + ``THROTTLER_SCOPES`` entry for the same scope takes precedence over the translated one. The deprecation warning is emitted by the downloader. """ scopes: dict[str, dict[str, Any]] = { scope_id: dict(config) - for scope_id, config in settings.getdict("THROTTLING_SCOPES").items() + for scope_id, config in settings.getdict("THROTTLER_SCOPES").items() } for slot_id, slot_config in settings.getdict("DOWNLOAD_SLOTS").items(): translated: dict[str, Any] = {} @@ -578,7 +578,7 @@ class ThrottlingManager: translated["delay"] = slot_config["delay"] if "randomize_delay" in slot_config: translated["jitter"] = 0.5 if slot_config["randomize_delay"] else 0.0 - # An explicit THROTTLING_SCOPES entry wins over the translated one. + # An explicit THROTTLER_SCOPES entry wins over the translated one. scopes[slot_id] = {**translated, **scopes.get(slot_id, {})} return scopes @@ -597,14 +597,14 @@ class ThrottlingManager: :func:`scope_cache`). Subclasses whose :meth:`get_scopes` cannot be resolved synchronously rely on that persisted value instead. """ - scopes = request.meta.get("throttling_scopes") + scopes = request.meta.get("throttler_scopes") if scopes is not None: return cast("RequestScopes", scopes) download_slot = request.meta.get("download_slot") if download_slot is not None: warnings.warn( "The 'download_slot' request meta key is deprecated. Use " - "'throttling_scopes' instead.", + "'throttler_scopes' instead.", category=ScrapyDeprecationWarning, stacklevel=2, ) @@ -641,7 +641,7 @@ class ThrottlingManager: # -- Scope-state coordination (called from the request lifecycle) -------- - def get_scope_manager(self, scope_id: ScopeID) -> ThrottlingScopeManagerProtocol: + def get_scope_manager(self, scope_id: ScopeID) -> ThrottlerScopeManagerProtocol: manager = self._scope_managers.get(scope_id) if manager is not None: # Mark as most-recently-used for the LRU scope limit. @@ -655,7 +655,7 @@ class ThrottlingManager: else self._default_scope_manager_cls ) manager = cast( - "ThrottlingScopeManagerProtocol", + "ThrottlerScopeManagerProtocol", build_from_crawler(manager_cls, self.crawler, config), ) self._scope_managers[scope_id] = manager @@ -664,7 +664,7 @@ class ThrottlingManager: def _enforce_scope_limit(self, keep: ScopeID) -> None: """Evict least-recently-used idle scopes while the number of live scope - managers exceeds :setting:`THROTTLING_SCOPE_LIMIT` (``0`` disables the + managers exceeds :setting:`THROTTLER_SCOPE_LIMIT` (``0`` disables the limit). LRU order is kept by :meth:`get_scope_manager` moving each accessed @@ -684,7 +684,7 @@ class ThrottlingManager: del self._scope_managers[scope_id] async def acquire(self, request: Request) -> None: - # A throttling-aware scheduler reserves the request before handing it + # A throttler-aware scheduler reserves the request before handing it # to the engine, so there is nothing left to wait for or record here. if request in self._reserved: return @@ -728,7 +728,7 @@ class ThrottlingManager: def _record_reservation( self, request: Request, - managers: list[tuple[ThrottlingScopeManagerProtocol, float | None]], + managers: list[tuple[ThrottlerScopeManagerProtocol, float | None]], ) -> None: """Record a send on each of *request*'s scope *managers* and mark *request* as reserved, so :meth:`release` can later free the slots. This @@ -744,7 +744,7 @@ class ThrottlingManager: for manager, _ in managers: manager.record_done() - # -- Synchronous readiness API (used by a throttling-aware scheduler) ------ + # -- Synchronous readiness API (used by a throttler-aware scheduler) ------ def is_ready(self, request: Request) -> bool: now = time.monotonic() @@ -759,7 +759,7 @@ class ThrottlingManager: return True def reserve(self, request: Request) -> None: - # A throttling-aware scheduler reserves every request before handing it + # A throttler-aware scheduler reserves every request before handing it # to the engine, so acquire() always returns early for it and never # gets to evict idle scopes; do it here so their managers do not pile # up on broad crawls. @@ -789,8 +789,8 @@ class ThrottlingManager: """Block until any of *managers* frees a concurrency slot. Each manager hands out an event Deferred that fires when a slot is freed - (via :meth:`ThrottlingScopeManager.record_done`) or the limit is raised - (via :meth:`ThrottlingScopeManager.set_concurrency`). A long safety timer + (via :meth:`ThrottlerScopeManager.record_done`) or the limit is raised + (via :meth:`ThrottlerScopeManager.set_concurrency`). A long safety timer bounds the wait in case no slot is ever freed (it always should be, via :meth:`release`). """ @@ -802,7 +802,7 @@ class ThrottlingManager: manager.discard_slot_event(event) async def _delay_request(self, request: Request) -> None: - """Honor the :reqmeta:`throttling_delay` meta key by holding *request* + """Honor the :reqmeta:`delay` meta key by holding *request* for the requested number of seconds the first time it is processed. This is the blocking (:meth:`acquire`) counterpart of @@ -815,31 +815,31 @@ class ThrottlingManager: if wait <= 0: return await sleep(wait) - request.meta["_throttling_delayed"] = True + request.meta["_throttler_delayed"] = True def _request_delay_deadline(self, request: Request, now: float) -> float: """Return the monotonic time before which *request* must not be sent due - to its :reqmeta:`throttling_delay`, or ``0.0`` if it has none. + to its :reqmeta:`delay`, or ``0.0`` if it has none. This is the readiness-API counterpart of :meth:`_delay_request`: - a throttling-aware scheduler gates requests through :meth:`is_ready` and + a throttler-aware scheduler gates requests through :meth:`is_ready` and :meth:`get_time_until_ready` instead of awaiting :meth:`acquire`, so the delay is enforced by holding back the request until this deadline rather than by sleeping. The deadline is computed once, the first time the request reaches the gate, and stored so later polls reuse it. - A request whose delay has already been honored (the ``_throttling_delayed`` + A request whose delay has already been honored (the ``_throttler_delayed`` flag, also set by :meth:`_delay_request`) is never delayed again, which keeps a resumed crawl from re-blocking on a stale deadline.""" - delay = request.meta.get("throttling_delay") - if not delay or request.meta.get("_throttling_delayed"): + delay = request.meta.get("delay") + if not delay or request.meta.get("_throttler_delayed"): return 0.0 - deadline = request.meta.get("_throttling_delay_deadline") + deadline = request.meta.get("_throttler_delay_deadline") if deadline is None: deadline = now + float(delay) - request.meta["_throttling_delay_deadline"] = deadline + request.meta["_throttler_delay_deadline"] = deadline if self._debug: - logger.debug(f"Holding {request} for {delay:.2f}s (throttling_delay)") + logger.debug(f"Holding {request} for {delay:.2f}s (delay)") return deadline def back_off( @@ -889,11 +889,11 @@ class ThrottlingManager: def apply_robots_crawl_delay(self, scope_id: ScopeID, delay: float) -> None: """Honor a robots.txt ``Crawl-delay`` directive of *delay* seconds for *scope_id* by setting its delay (capped at - :setting:`THROTTLING_ROBOTSTXT_MAX_DELAY`) and its concurrency to ``1``. + :setting:`THROTTLER_ROBOTSTXT_MAX_DELAY`) and its concurrency to ``1``. Called from the :signal:`robots_parsed` signal handler when - :setting:`THROTTLING_ROBOTSTXT_OBEY` is enabled. An explicit - :setting:`THROTTLING_SCOPES` configuration for the scope is respected, but + :setting:`THROTTLER_ROBOTSTXT_OBEY` is enabled. An explicit + :setting:`THROTTLER_SCOPES` configuration for the scope is respected, but a warning is logged about the discrepancy unless its ``ignore_robots_txt`` key is ``True``. """ @@ -910,10 +910,10 @@ class ThrottlingManager: conflicts.append(f"concurrency={config['concurrency']!r} > 1") if conflicts: logger.warning( - f"Throttling scope {scope_id!r} is configured with {' and '.join(conflicts)}, " + f"Throttler scope {scope_id!r} is configured with {' and '.join(conflicts)}, " f"which is more aggressive than its robots.txt Crawl-delay of " f"{capped}s. The configured values take precedence; set " - "'ignore_robots_txt': True in its THROTTLING_SCOPES entry to " + "'ignore_robots_txt': True in its THROTTLER_SCOPES entry to " "silence this warning." ) return @@ -951,12 +951,12 @@ class ThrottlingManager: del self._scope_managers[scope_id] -class ThrottlingScopeManagerProtocol(Protocol): - """A protocol for :setting:`THROTTLING_SCOPE_MANAGER` :ref:`components +class ThrottlerScopeManagerProtocol(Protocol): + """A protocol for :setting:`THROTTLER_SCOPE_MANAGER` :ref:`components `. The ``__init__`` method gets a ``config`` dict with the base configuration - of the managed throttling scope. For example: + of the managed throttler scope. For example: .. code-block:: python @@ -988,7 +988,7 @@ class ThrottlingScopeManagerProtocol(Protocol): """Return the number of seconds to wait before a request for this scope may be sent, or ``0`` if it may be sent right away. - *amount* is the expected :ref:`throttling quota ` + *amount* is the expected :ref:`throttler quota ` consumption of the request, if any. """ @@ -996,7 +996,7 @@ class ThrottlingScopeManagerProtocol(Protocol): self, now: float | None = None, amount: float | None = None ) -> None: """Record that a request for this scope has just been sent, consuming - *amount* of its :ref:`throttling quota ` if given.""" + *amount* of its :ref:`throttler quota ` if given.""" def record_done(self, now: float | None = None) -> None: """Record that a previously :meth:`record_sent` request has finished @@ -1017,7 +1017,7 @@ class ThrottlingScopeManagerProtocol(Protocol): *cap* limits *delay* to :setting:`BACKOFF_MAX_DELAY`. It is ``True`` for untrusted input such as response headers, and may be set to ``False`` for trusted, programmatic delays (see - :meth:`ThrottlingManagerProtocol.delay_scope`). + :meth:`ThrottlerProtocol.delay_scope`). """ def reconcile_quota( @@ -1026,7 +1026,7 @@ class ThrottlingScopeManagerProtocol(Protocol): remaining: float | None = None, now: float | None = None, ) -> None: - """Reconcile the :ref:`throttling quota ` of this + """Reconcile the :ref:`throttler quota ` of this scope with the actual *consumed* amount (or the *remaining* amount) reported for a request, correcting the estimate used by :meth:`record_sent`.""" @@ -1049,7 +1049,7 @@ class ThrottlingScopeManagerProtocol(Protocol): def concurrency_blocked(self) -> bool: """Return whether this scope is at its concurrency limit. - :class:`ThrottlingManager` calls this (after all time-based gates in + :class:`Throttler` calls this (after all time-based gates in :meth:`can_send` are open) to decide whether to wait for a freed slot. Return ``False`` when no concurrency limit is enforced. """ @@ -1058,7 +1058,7 @@ class ThrottlingScopeManagerProtocol(Protocol): """Return the current load of this scope: a non-negative number, with ``1.0`` meaning "as busy as its concurrency limit allows". - A :ref:`throttling-aware scheduler ` uses + A :ref:`throttler-aware scheduler ` uses this to break ties between equally-prioritized requests, preferring the least-loaded scopes. The reference implementation returns active sends divided by the concurrency limit (falling back to @@ -1076,7 +1076,7 @@ class ThrottlingScopeManagerProtocol(Protocol): def discard_slot_event(self, event: Deferred[None]) -> None: """Cancel a pending slot event returned by :meth:`slot_event`. - Called by :class:`ThrottlingManager` when the wait ends without the + Called by :class:`Throttler` when the wait ends without the event firing (e.g. another scope's slot opened first). """ @@ -1095,8 +1095,8 @@ class ThrottlingScopeManagerProtocol(Protocol): _SLOT_WAIT_TIMEOUT = 1.0 -class ThrottlingScopeManager: - """The default :setting:`THROTTLING_SCOPE_MANAGER` class. +class ThrottlerScopeManager: + """The default :setting:`THROTTLER_SCOPE_MANAGER` class. It implements a per-scope state machine covering delay, exponential :ref:`backoff `, concurrency and :ref:`quotas @@ -1122,7 +1122,7 @@ class ThrottlingScopeManager: than that many requests are allowed in flight at once. - When the scope is configured with a ``"quota"``, no more than that much - quota is consumed per ``"window"`` (default: :setting:`THROTTLING_WINDOW`). + quota is consumed per ``"window"`` (default: :setting:`THROTTLER_WINDOW`). """ @classmethod @@ -1134,7 +1134,7 @@ class ThrottlingScopeManager: backoff: dict[str, Any] = config.get("backoff", {}) self._id: ScopeID = config.get("id", "") # The per-scope delay defaults to DOWNLOAD_DELAY; a scope can override - # it with its own "delay" config (see THROTTLING_SCOPES). + # it with its own "delay" config (see THROTTLER_SCOPES). self._base_delay: float = float( config.get("delay", settings.getfloat("DOWNLOAD_DELAY")) ) @@ -1180,7 +1180,7 @@ class ThrottlingScopeManager: quota = config.get("quota") self._quota: float | None = None if quota is None else float(quota) self._quota_window: float = float( - config.get("window", settings.getfloat("THROTTLING_WINDOW")) + config.get("window", settings.getfloat("THROTTLER_WINDOW")) ) # State. diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py index 4b69733dc..4ac67da50 100644 --- a/scrapy/utils/test.py +++ b/scrapy/utils/test.py @@ -75,9 +75,9 @@ def get_crawler( if prevent_warnings: # Pin the per-scope concurrency to CONCURRENT_REQUESTS_PER_DOMAIN's # default: keeps the suite's historical behavior and silences the - # warn-then-flip warning (see throttling._warn_on_deprecated_concurrency). + # warn-then-flip warning (see throttler._warn_on_deprecated_concurrency). settings.setdefault( - "THROTTLING_SCOPE_CONCURRENCY", + "THROTTLER_SCOPE_CONCURRENCY", default_settings.CONCURRENT_REQUESTS_PER_DOMAIN, ) runner: CrawlerRunnerBase diff --git a/tests/test_crawler.py b/tests/test_crawler.py index 578b2396a..9b1264e76 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -55,12 +55,12 @@ def get_raw_crawler( class TestScopeConcurrencyBridge: """CONCURRENT_REQUESTS_PER_DOMAIN is deprecated and bridged into - THROTTLING_SCOPE_CONCURRENCY by setting priority (see - scrapy.throttling._default_scope_concurrency).""" + THROTTLER_SCOPE_CONCURRENCY by setting priority (see + scrapy.throttler._default_scope_concurrency).""" @staticmethod def _resolve(settings: Settings) -> int: - from scrapy.throttling import _default_scope_concurrency # noqa: PLC0415 + from scrapy.throttler import _default_scope_concurrency # noqa: PLC0415 return _default_scope_concurrency(settings) @@ -75,18 +75,18 @@ class TestScopeConcurrencyBridge: def test_scope_concurrency_overrides(self) -> None: settings = Settings() - settings.set("THROTTLING_SCOPE_CONCURRENCY", 4, priority="project") + settings.set("THROTTLER_SCOPE_CONCURRENCY", 4, priority="project") assert self._resolve(settings) == 4 def test_new_setting_wins_on_non_default_tie(self) -> None: settings = Settings() settings.set("CONCURRENT_REQUESTS_PER_DOMAIN", 4, priority="project") - settings.set("THROTTLING_SCOPE_CONCURRENCY", 9, priority="project") + settings.set("THROTTLER_SCOPE_CONCURRENCY", 9, priority="project") assert self._resolve(settings) == 9 def test_higher_priority_deprecated_setting_wins(self) -> None: settings = Settings() - settings.set("THROTTLING_SCOPE_CONCURRENCY", 9, priority="project") + settings.set("THROTTLER_SCOPE_CONCURRENCY", 9, priority="project") settings.set("CONCURRENT_REQUESTS_PER_DOMAIN", 4, priority="cmdline") assert self._resolve(settings) == 4 @@ -111,7 +111,7 @@ class TestScopeConcurrencyBridge: with warnings.catch_warnings(record=True) as recorded: warnings.simplefilter("always") get_crawler( - settings_dict={"THROTTLING_SCOPE_CONCURRENCY": 4}, + settings_dict={"THROTTLER_SCOPE_CONCURRENCY": 4}, prevent_warnings=False, ) messages = [str(w.message) for w in recorded] diff --git a/tests/test_engine.py b/tests/test_engine.py index 4bca525c5..e4f44ca0a 100644 --- a/tests/test_engine.py +++ b/tests/test_engine.py @@ -655,7 +655,7 @@ async def test_request_scheduled_signal(): crawler.signals.disconnect(signal_handler, signals.request_scheduled) -class TestEngineThrottling: +class TestEngineThrottler: @pytest.fixture def engine(self): crawler = get_crawler(MySpider) @@ -708,25 +708,25 @@ class TestEngineThrottling: engine._maybe_arm_delay_wakeup() assert engine._delay_wakeup is None - def test_maybe_warn_throttling_backout(self, engine): - # A scheduler without get_next_request_delay is not throttling-aware, so + def test_maybe_warn_throttler_backout(self, engine): + # A scheduler without get_next_request_delay is not throttler-aware, so # the warning recommends switching to one. engine._get_next_request_delay = None with LogCapture() as log: - engine._maybe_warn_throttling_backout() + engine._maybe_warn_throttler_backout() # A second call is a no-op (the warning is emitted only once). - engine._maybe_warn_throttling_backout() - assert engine._throttling_backout_warned is True - assert str(log).count("ThrottlingAwareScheduler") == 1 + engine._maybe_warn_throttler_backout() + assert engine._throttler_backout_warned is True + assert str(log).count("ThrottlerAwareScheduler") == 1 - def test_maybe_warn_throttling_backout_throttling_aware(self, engine): - # A throttling-aware scheduler (one with get_next_request_delay) holds + def test_maybe_warn_throttler_backout_throttler_aware(self, engine): + # A throttler-aware scheduler (one with get_next_request_delay) holds # throttled requests itself, so no warning is emitted. engine._get_next_request_delay = lambda: None with LogCapture() as log: - engine._maybe_warn_throttling_backout() - assert engine._throttling_backout_warned is False - assert "ThrottlingAwareScheduler" not in str(log) + engine._maybe_warn_throttler_backout() + assert engine._throttler_backout_warned is False + assert "ThrottlerAwareScheduler" not in str(log) def test_spider_is_idle_false_while_scheduling(self, engine): engine._slot = Mock() @@ -734,7 +734,7 @@ class TestEngineThrottling: engine.scraper.slot.is_idle.return_value = True engine.downloader = Mock() engine.downloader.active = [] - engine._throttling_waiting = set() + engine._throttler_waiting = set() engine._start = None engine._scheduling = 1 # An in-flight async enqueue keeps the spider from being considered idle. diff --git a/tests/test_pqueues.py b/tests/test_pqueues.py index cf7f94f04..cff66bb1e 100644 --- a/tests/test_pqueues.py +++ b/tests/test_pqueues.py @@ -8,11 +8,11 @@ from scrapy.http.request import Request from scrapy.pqueues import ( DownloaderAwarePriorityQueue, ScrapyPriorityQueue, - ThrottlingAwarePriorityQueue, + ThrottlerAwarePriorityQueue, ) from scrapy.spiders import Spider from scrapy.squeues import FifoMemoryQueue, PickleFifoDiskQueue -from scrapy.throttling import iter_scopes +from scrapy.throttler import iter_scopes from scrapy.utils.misc import build_from_crawler, load_object from scrapy.utils.test import get_crawler from tests.test_scheduler import MockDownloader @@ -191,22 +191,22 @@ class TestDownloaderAwarePriorityQueue: # No active downloads are tracked in the downloader, so every slot has # the same score and tie-breaking must not starve a slot. req_a1 = Request("https://example.org/a1") - req_a1.meta["throttling_scopes"] = "slot-a" + req_a1.meta["throttler_scopes"] = "slot-a" req_b1 = Request("https://example.org/b1") - req_b1.meta["throttling_scopes"] = "slot-b" + req_b1.meta["throttler_scopes"] = "slot-b" req_a2 = Request("https://example.org/a2") - req_a2.meta["throttling_scopes"] = "slot-a" + req_a2.meta["throttler_scopes"] = "slot-a" req_b2 = Request("https://example.org/b2") - req_b2.meta["throttling_scopes"] = "slot-b" + req_b2.meta["throttler_scopes"] = "slot-b" for request in (req_a1, req_b1, req_a2, req_b2): self.queue.push(request) slots = [ - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], ] assert slots == ["slot-a", "slot-b", "slot-a", "slot-b"] @@ -215,22 +215,22 @@ class TestDownloaderAwarePriorityQueue: # If the selected slot becomes empty, rotation should continue from # that slot marker to avoid restarting from the smallest slot. req_a1 = Request("https://example.org/a1") - req_a1.meta["throttling_scopes"] = "slot-a" + req_a1.meta["throttler_scopes"] = "slot-a" req_a2 = Request("https://example.org/a2") - req_a2.meta["throttling_scopes"] = "slot-a" + req_a2.meta["throttler_scopes"] = "slot-a" req_b1 = Request("https://example.org/b1") - req_b1.meta["throttling_scopes"] = "slot-b" + req_b1.meta["throttler_scopes"] = "slot-b" req_c1 = Request("https://example.org/c1") - req_c1.meta["throttling_scopes"] = "slot-c" + req_c1.meta["throttler_scopes"] = "slot-c" for request in (req_a1, req_a2, req_b1, req_c1): self.queue.push(request) slots = [ - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], - self.queue.pop().meta["throttling_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], + self.queue.pop().meta["throttler_scopes"], ] assert slots == ["slot-a", "slot-b", "slot-c", "slot-a"] @@ -240,11 +240,11 @@ class TestDownloaderAwarePriorityQueue: assert throttler is not None req_a = Request("https://example.org/a") - req_a.meta["throttling_scopes"] = "slot-a" + req_a.meta["throttler_scopes"] = "slot-a" req_b = Request("https://example.org/b") - req_b.meta["throttling_scopes"] = "slot-b" + req_b.meta["throttler_scopes"] = "slot-b" req_c = Request("https://example.org/c") - req_c.meta["throttling_scopes"] = "slot-c" + req_c.meta["throttler_scopes"] = "slot-c" for req in (req_a, req_b, req_c): self.queue.push(req) @@ -257,7 +257,7 @@ class TestDownloaderAwarePriorityQueue: def test_contains(self): req = Request("https://example.org/") - req.meta["throttling_scopes"] = "example-slot" + req.meta["throttler_scopes"] = "example-slot" assert "example-slot" not in self.queue self.queue.push(req) assert "example-slot" in self.queue @@ -317,10 +317,10 @@ def test_pop_order(input_, output): assert actual_output_urls == expected_output_urls -class TestThrottlingAwarePriorityQueue: +class TestThrottlerAwarePriorityQueue: def _queue(self, crawler, key=""): return build_from_crawler( - ThrottlingAwarePriorityQueue, + ThrottlerAwarePriorityQueue, crawler, downstream_queue_cls=FifoMemoryQueue, key=key, @@ -346,7 +346,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler( Spider, settings_dict={ - "THROTTLING_SCOPES": {"slow.com": {"delay": 1000.0}}, + "THROTTLER_SCOPES": {"slow.com": {"delay": 1000.0}}, "RANDOMIZE_DOWNLOAD_DELAY": False, }, ) @@ -369,13 +369,13 @@ class TestThrottlingAwarePriorityQueue: assert delay == pytest.approx(1000.0, abs=1.0) @coroutine_test - async def test_pop_holds_request_with_throttling_delay(self): + async def test_pop_holds_request_with_delay(self): crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False}) queue = self._queue(crawler) await self._push( queue, crawler, - Request("http://slow.com/1", meta={"throttling_delay": 1000.0}), + Request("http://slow.com/1", meta={"delay": 1000.0}), ) await self._push(queue, crawler, Request("http://fast.com/1")) # The delayed request is held back even though its scope is otherwise @@ -398,7 +398,7 @@ class TestThrottlingAwarePriorityQueue: await self._push( queue, crawler, - Request("http://example.com/slow", meta={"throttling_delay": 1000.0}), + Request("http://example.com/slow", meta={"delay": 1000.0}), ) await self._push(queue, crawler, Request("http://example.com/fast")) # The delayed request is held aside, so the other request in the same @@ -413,7 +413,7 @@ class TestThrottlingAwarePriorityQueue: async def test_delayed_request_promoted_when_due(self): crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False}) queue = self._queue(crawler) - request = Request("http://example.com/slow", meta={"throttling_delay": 1000.0}) + request = Request("http://example.com/slow", meta={"delay": 1000.0}) await self._push(queue, crawler, request) assert queue.pop() is None # held back by its per-request delay # Once the delay elapses the request is promoted into its scope-set @@ -421,7 +421,7 @@ class TestThrottlingAwarePriorityQueue: queue._promote_ready(queue._delayed[0][0]) popped = queue.pop() assert popped is request - assert popped.meta["_throttling_delayed"] is True + assert popped.meta["_throttler_delayed"] is True assert len(queue) == 0 @coroutine_test @@ -432,7 +432,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False}) temp_dir = tempfile.mkdtemp() queue = build_from_crawler( - ThrottlingAwarePriorityQueue, + ThrottlerAwarePriorityQueue, crawler, downstream_queue_cls=PickleFifoDiskQueue, key=temp_dir, @@ -440,13 +440,13 @@ class TestThrottlingAwarePriorityQueue: await self._push( queue, crawler, - Request("http://example.com/slow", meta={"throttling_delay": 1000.0}), + Request("http://example.com/slow", meta={"delay": 1000.0}), ) assert len(queue) == 1 # held in memory, not yet in any scope-set queue state = queue.close() # graceful stop resumed = build_from_crawler( - ThrottlingAwarePriorityQueue, + ThrottlerAwarePriorityQueue, crawler, downstream_queue_cls=PickleFifoDiskQueue, key=temp_dir, @@ -457,7 +457,7 @@ class TestThrottlingAwarePriorityQueue: assert popped is not None assert popped.url == "http://example.com/slow" # Its delay is marked consumed, so it does not re-block on resume. - assert popped.meta["_throttling_delayed"] is True + assert popped.meta["_throttler_delayed"] is True resumed.close() @coroutine_test @@ -465,7 +465,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler( Spider, settings_dict={ - "THROTTLING_SCOPES": { + "THROTTLER_SCOPES": { "a.com": {"concurrency": 4}, "b.com": {"concurrency": 4}, } @@ -486,7 +486,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler( Spider, settings_dict={ - "THROTTLING_SCOPES": { + "THROTTLER_SCOPES": { "a.com": {"concurrency": 4}, "b.com": {"concurrency": 4}, } @@ -512,18 +512,6 @@ class TestThrottlingAwarePriorityQueue: await self._push(queue, crawler, Request("http://a.com/1")) assert queue.close() != {} - @coroutine_test - async def test_peek(self): - crawler = get_crawler(Spider) - queue = self._queue(crawler) - assert queue.peek() is None - await self._push(queue, crawler, Request("http://a.com/1")) - head = queue.peek() - assert head is not None - assert head.url == "http://a.com/1" - # peek() does not consume the request. - assert len(queue) == 1 - @coroutine_test async def test_get_next_request_delay_zero_when_ready(self): crawler = get_crawler(Spider) @@ -537,7 +525,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler(Spider) queue = self._queue(crawler) # An empty (but still registered) internal queue is skipped. - queue.pqueues[frozenset({"a.com"})] = queue.pqfactory(frozenset({"a.com"})) + queue.pqueues[frozenset({"a.com"})] = queue._pqfactory(frozenset({"a.com"})) assert queue.get_next_request_delay() is None @coroutine_test @@ -545,7 +533,7 @@ class TestThrottlingAwarePriorityQueue: crawler = get_crawler( Spider, settings_dict={ - "THROTTLING_SCOPES": { + "THROTTLER_SCOPES": { "a.com": {"delay": 10.0}, "b.com": {"delay": 1000.0}, }, @@ -572,27 +560,16 @@ class TestThrottlingAwarePriorityQueue: queue = self._queue(crawler) await self._push(queue, crawler, Request("http://a.com/1")) inner = next(iter(queue.pqueues.values())) - # peek() still reports a sendable head, but pop() yields nothing: the + # _select() still reports a sendable head, but pop() yields nothing: the # request-is-None guard must not try to reserve a missing request. inner.pop = lambda: None assert queue.pop() is None - @coroutine_test - async def test_contains(self): - crawler = get_crawler(Spider) - queue = self._queue(crawler) - scope_set = frozenset( - iter_scopes(await crawler.throttler.get_scopes(Request("http://a.com/1"))) - ) - assert scope_set not in queue - await self._push(queue, crawler, Request("http://a.com/1")) - assert scope_set in queue - def test_non_dict_slot_startprios(self): crawler = get_crawler(Spider) with pytest.raises(ValueError, match="slot_startprios"): build_from_crawler( - ThrottlingAwarePriorityQueue, + ThrottlerAwarePriorityQueue, crawler, downstream_queue_cls=FifoMemoryQueue, key="", diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py index b8b1f8c7d..05dda6026 100644 --- a/tests/test_scheduler.py +++ b/tests/test_scheduler.py @@ -11,7 +11,7 @@ from unittest.mock import Mock import pytest from scrapy.core.downloader import Downloader -from scrapy.core.scheduler import BaseScheduler, Scheduler, ThrottlingAwareScheduler +from scrapy.core.scheduler import BaseScheduler, Scheduler, ThrottlerAwareScheduler from scrapy.crawler import Crawler from scrapy.exceptions import ScrapyDeprecationWarning from scrapy.http import Request @@ -411,22 +411,22 @@ class TestIncompatibility: self._incompatible() -_THROTTLING_AWARE_PQ = "scrapy.pqueues.ThrottlingAwarePriorityQueue" +_THROTTLER_AWARE_PQ = "scrapy.pqueues.ThrottlerAwarePriorityQueue" -class TestThrottlingAwareScheduler: +class TestThrottlerAwareScheduler: def _crawler(self, settings_dict: dict[str, Any] | None = None) -> Crawler: settings = { - "SCHEDULER_PRIORITY_QUEUE": _THROTTLING_AWARE_PQ, + "SCHEDULER_PRIORITY_QUEUE": _THROTTLER_AWARE_PQ, "DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter", **(settings_dict or {}), } return get_crawler(Spider, settings) - def _scheduler(self, crawler: Crawler) -> ThrottlingAwareScheduler: + def _scheduler(self, crawler: Crawler) -> ThrottlerAwareScheduler: spider = Spider(name="spider") crawler.spider = spider - scheduler = ThrottlingAwareScheduler.from_crawler(crawler) + scheduler = ThrottlerAwareScheduler.from_crawler(crawler) scheduler.open(spider) return scheduler @@ -449,21 +449,21 @@ class TestThrottlingAwareScheduler: scheduler.enqueue_request(Request("http://a.com/1")) scheduler.close("finished") - def test_requires_throttling_aware_priority_queue(self) -> None: + def test_requires_throttler_aware_priority_queue(self) -> None: crawler = self._crawler( {"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ScrapyPriorityQueue"} ) spider = Spider(name="spider") crawler.spider = spider - scheduler = ThrottlingAwareScheduler.from_crawler(crawler) - with pytest.raises(ValueError, match="throttling-aware priority queue"): + scheduler = ThrottlerAwareScheduler.from_crawler(crawler) + with pytest.raises(ValueError, match="throttler-aware priority queue"): scheduler.open(spider) @coroutine_test async def test_delay_blocks_and_reports_delay(self) -> None: crawler = self._crawler( { - "THROTTLING_SCOPES": {"slow.com": {"delay": 1000.0}}, + "THROTTLER_SCOPES": {"slow.com": {"delay": 1000.0}}, "RANDOMIZE_DOWNLOAD_DELAY": False, }, ) @@ -481,7 +481,7 @@ class TestThrottlingAwareScheduler: @coroutine_test async def test_no_delay_when_only_concurrency_blocked(self) -> None: crawler = self._crawler( - {"THROTTLING_SCOPES": {"slow.com": {"concurrency": 1}}}, + {"THROTTLER_SCOPES": {"slow.com": {"concurrency": 1}}}, ) scheduler = self._scheduler(crawler) await scheduler.enqueue_request_async(Request("http://slow.com/1")) @@ -494,14 +494,14 @@ class TestThrottlingAwareScheduler: @coroutine_test async def test_delayed_request_survives_jobdir_stop(self, tmp_path: Path) -> None: - # A request held back by its per-request throttling_delay must not be + # A request held back by its per-request delay must not be # lost on a graceful stop when a JOBDIR is configured: it is flushed to # the disk queue on close and restored on resume. crawler = self._crawler( {"JOBDIR": str(tmp_path), "RANDOMIZE_DOWNLOAD_DELAY": False} ) scheduler = self._scheduler(crawler) - request = Request("http://a.com/slow", meta={"throttling_delay": 1000.0}) + request = Request("http://a.com/slow", meta={"delay": 1000.0}) assert await scheduler.enqueue_request_async(request) is True assert len(scheduler) == 1 # The delay holds it back, so nothing is dequeued before the stop. @@ -585,14 +585,14 @@ class TestThrottlingAwareScheduler: scheduler2.close("finished") -class TestIntegrationWithThrottlingAwareScheduler: +class TestIntegrationWithThrottlerAwareScheduler: @inline_callbacks_test def test_integration(self): crawler = get_crawler( spidercls=StartUrlsSpider, settings_dict={ - "SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler", - "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue", + "SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler", + "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue", "DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter", }, ) @@ -628,8 +628,8 @@ class TestIntegrationWithThrottlingAwareScheduler: crawler = get_crawler( spidercls=FollowSpider, settings_dict={ - "SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler", - "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue", + "SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler", + "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue", "DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter", }, ) @@ -638,16 +638,16 @@ class TestIntegrationWithThrottlingAwareScheduler: @inline_callbacks_test def test_integration_with_delay(self): - # A small per-scope delay forces the engine to arm the throttling wakeup + # A small per-scope delay forces the engine to arm the throttler wakeup # timer between requests; the crawl must still complete. crawler = get_crawler( spidercls=StartUrlsSpider, settings_dict={ - "SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler", - "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue", + "SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler", + "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue", "DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter", "RANDOMIZE_DOWNLOAD_DELAY": False, - "THROTTLING_SCOPES": {"127.0.0.1": {"delay": 0.05}}, + "THROTTLER_SCOPES": {"127.0.0.1": {"delay": 0.05}}, }, ) with MockServer() as mockserver: diff --git a/tests/test_throttling.py b/tests/test_throttler.py similarity index 90% rename from tests/test_throttling.py rename to tests/test_throttler.py index b05181a29..503ecc26d 100644 --- a/tests/test_throttling.py +++ b/tests/test_throttler.py @@ -8,9 +8,9 @@ from scrapy import signals from scrapy.exceptions import DownloadTimeoutError from scrapy.http import Request, Response from scrapy.settings import default_settings -from scrapy.throttling import ( - ThrottlingManager, - ThrottlingScopeManager, +from scrapy.throttler import ( + Throttler, + ThrottlerScopeManager, _add_bare_scope, _parse_ratelimit_reset, _parse_retry_after, @@ -30,12 +30,12 @@ from tests.utils.decorators import coroutine_test def _manager(settings=None): crawler = get_crawler(settings_dict=settings) - return ThrottlingManager.from_crawler(crawler) + return Throttler.from_crawler(crawler) def _scope_manager(settings=None, config=None): crawler = get_crawler(settings_dict=settings) - return ThrottlingScopeManager.from_crawler(crawler, config or {"id": "example.com"}) + return ThrottlerScopeManager.from_crawler(crawler, config or {"id": "example.com"}) class _FakeRobotParser: @@ -67,11 +67,11 @@ def test_deprecated_concurrency_defaults_differ(): shipping a bogus warning or aborting a crawl.""" assert ( default_settings.CONCURRENT_REQUESTS_PER_DOMAIN - != default_settings.THROTTLING_SCOPE_CONCURRENCY + != default_settings.THROTTLER_SCOPE_CONCURRENCY ) -class TestThrottlingManager: +class TestThrottler: @coroutine_test async def test_get_scopes_returns_netloc(self): manager = _manager() @@ -91,20 +91,20 @@ class TestThrottlingManager: @coroutine_test async def test_get_scopes_meta_string(self): manager = _manager() - request = Request("http://example.com/a", meta={"throttling_scopes": "api"}) + request = Request("http://example.com/a", meta={"throttler_scopes": "api"}) assert await manager.get_scopes(request) == "api" @coroutine_test async def test_get_scopes_meta_dict(self): manager = _manager() request = Request( - "http://example.com/a", meta={"throttling_scopes": {"api": 2.0}} + "http://example.com/a", meta={"throttler_scopes": {"api": 2.0}} ) assert await manager.get_scopes(request) == {"api": 2.0} @coroutine_test async def test_get_scopes_persisted_in_meta(self): - from scrapy.throttling import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415 + from scrapy.throttler import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415 manager = _manager() request = Request("http://example.com/a") @@ -114,7 +114,7 @@ class TestThrottlingManager: @coroutine_test async def test_scope_cache_works_without_crawler(self): # scope_cache only persists to meta; it needs nothing from the manager. - from scrapy.throttling import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415 + from scrapy.throttler import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415 class CrawlerlessManager: @scope_cache @@ -131,7 +131,7 @@ class TestThrottlingManager: # path reuses them instead of resolving again. calls = [] - class CountingManager(ThrottlingManager): + class CountingManager(Throttler): @scope_cache async def get_scopes(self, request): calls.append(request.url) @@ -155,7 +155,7 @@ class TestThrottlingManager: manager = _manager() request = Request( - "http://example.com/a", meta={"throttling_scopes": {"bucket": 3.0}} + "http://example.com/a", meta={"throttler_scopes": {"bucket": 3.0}} ) await manager.get_scopes(request) # A request restored from a disk queue is a fresh object; the synchronous @@ -183,13 +183,13 @@ class TestThrottlingManager: def test_scope_manager_class_in_config(self): manager = _manager( - {"THROTTLING_SCOPES": {"example.com": {"manager": ThrottlingScopeManager}}} + {"THROTTLER_SCOPES": {"example.com": {"manager": ThrottlerScopeManager}}} ) scope = manager._get_scope_manager("example.com") - assert isinstance(scope, ThrottlingScopeManager) + assert isinstance(scope, ThrottlerScopeManager) def test_release_frees_concurrency(self): - manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}}) + manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}}) scope = manager._get_scope_manager("example.com") request = Request("http://example.com") scope.record_sent(now=0.0) @@ -205,7 +205,7 @@ class TestThrottlingManager: async def test_acquire_waits_for_freed_slot(self): from scrapy.utils.asyncio import call_later # noqa: PLC0415 - manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}}) + manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}}) r1 = Request("http://example.com/1") r2 = Request("http://example.com/2") # Drive acquire() the way the engine does, so it runs as a real task that @@ -222,7 +222,7 @@ class TestThrottlingManager: assert r2 in manager._reserved def test_apply_backoff_reconciles_quota_without_backoff(self): - manager = _manager({"THROTTLING_SCOPES": {"cost": {"quota": 100.0}}}) + manager = _manager({"THROTTLER_SCOPES": {"cost": {"quota": 100.0}}}) scope = manager._get_scope_manager("cost") manager._apply_backoff({"cost": {"consumed": 5.0}}) # Quota was reconciled but no backoff step was applied. @@ -230,7 +230,7 @@ class TestThrottlingManager: assert scope._backoff_level == 0 def test_apply_backoff_delay_and_consumed(self): - manager = _manager({"THROTTLING_SCOPES": {"cost": {"quota": 100.0}}}) + manager = _manager({"THROTTLER_SCOPES": {"cost": {"quota": 100.0}}}) scope = manager._get_scope_manager("cost") manager._apply_backoff({"cost": {"delay": 5.0, "consumed": 2.0}}) assert scope._consumed == pytest.approx(2.0) @@ -317,7 +317,7 @@ class TestThrottlingManager: ("settings", "parser_delay", "expected_base_delay"), [ ({"ROBOTSTXT_OBEY": True, "RANDOMIZE_DOWNLOAD_DELAY": False}, 3.0, 3.0), - ({"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_OBEY": False}, 3.0, 0.0), + ({"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_OBEY": False}, 3.0, 0.0), ({"ROBOTSTXT_OBEY": True}, None, 0.0), ({"ROBOTSTXT_OBEY": True}, ValueError(), 0.0), ], @@ -344,13 +344,13 @@ class TestThrottlingManager: def test_apply_robots_crawl_delay_capped(self): manager = _manager( - {"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_MAX_DELAY": 2.0} + {"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_MAX_DELAY": 2.0} ) manager.apply_robots_crawl_delay("example.com", 30.0) assert manager._get_scope_manager("example.com")._base_delay == 2.0 def test_apply_robots_crawl_delay_disabled(self): - manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_OBEY": False}) + manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_OBEY": False}) manager.apply_robots_crawl_delay("example.com", 3.0) assert manager._get_scope_manager("example.com")._base_delay == 0.0 @@ -363,10 +363,10 @@ class TestThrottlingManager: manager = _manager( { "ROBOTSTXT_OBEY": True, - "THROTTLING_SCOPES": {"example.com": {"concurrency": 8}}, + "THROTTLER_SCOPES": {"example.com": {"concurrency": 8}}, } ) - with caplog.at_level(logging.WARNING, logger="scrapy.throttling"): + with caplog.at_level(logging.WARNING, logger="scrapy.throttler"): manager.apply_robots_crawl_delay("example.com", 3.0) assert "Crawl-delay" in caplog.text # The configured value takes precedence (crawl-delay not applied). @@ -376,19 +376,19 @@ class TestThrottlingManager: manager = _manager( { "ROBOTSTXT_OBEY": True, - "THROTTLING_SCOPES": { + "THROTTLER_SCOPES": { "example.com": {"concurrency": 8, "ignore_robots_txt": True} }, } ) - with caplog.at_level(logging.WARNING, logger="scrapy.throttling"): + with caplog.at_level(logging.WARNING, logger="scrapy.throttler"): manager.apply_robots_crawl_delay("example.com", 3.0) # No warning is logged and the crawl-delay is not applied. assert "Crawl-delay" not in caplog.text assert manager._get_scope_manager("example.com")._base_delay == 0.0 def test_scope_eviction(self): - manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 100.0}) + manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 100.0}) scope = manager._get_scope_manager("example.com") scope.record_sent(now=0.0) scope.record_done(now=0.0) @@ -403,22 +403,22 @@ class TestThrottlingManager: def test_scope_eviction_skips_active_backoff(self): manager = _manager( - {"THROTTLING_SCOPE_MAX_IDLE": 100.0, "BACKOFF_MAX_DELAY": 100_000.0} + {"THROTTLER_SCOPE_MAX_IDLE": 100.0, "BACKOFF_MAX_DELAY": 100_000.0} ) scope = manager._get_scope_manager("example.com") scope.record_backoff(delay=10_000.0, now=0.0) manager._last_eviction = None manager._maybe_evict(now=5_000.0) # Still in backoff (in_backoff_until far in the future), so not evicted - # even though it has been idle for longer than THROTTLING_SCOPE_MAX_IDLE. + # even though it has been idle for longer than THROTTLER_SCOPE_MAX_IDLE. assert "example.com" in manager._scope_managers def test_reserve_evicts_idle_scopes(self): - # A throttling-aware scheduler reserves every request before the engine + # A throttler-aware scheduler reserves every request before the engine # reaches acquire() (which fast-paths reserved requests), so reserve() # must be the hook that evicts idle scope managers; otherwise they pile # up unbounded on broad crawls. - manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 1.0}) + manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 1.0}) idle = manager._get_scope_manager("idle.example") # Make it look long-idle: a finished send in the distant monotonic past. idle.record_sent(now=0.0) @@ -429,7 +429,7 @@ class TestThrottlingManager: assert "active.example" in manager._scope_managers def test_scope_limit_evicts_least_recently_used(self): - manager = _manager({"THROTTLING_SCOPE_LIMIT": 2}) + manager = _manager({"THROTTLER_SCOPE_LIMIT": 2}) # Use three scopes in order; each send/done leaves them idle. for scope_id in ("a.example", "b.example", "c.example"): scope = manager._get_scope_manager(scope_id) @@ -439,7 +439,7 @@ class TestThrottlingManager: assert set(manager._scope_managers) == {"b.example", "c.example"} def test_scope_limit_keeps_active_scopes(self): - manager = _manager({"THROTTLING_SCOPE_LIMIT": 1}) + manager = _manager({"THROTTLER_SCOPE_LIMIT": 1}) # Two scopes with in-flight requests cannot be evicted, so the limit is # exceeded rather than dropping a scope that still tracks a live send. for scope_id in ("a.example", "b.example"): @@ -447,7 +447,7 @@ class TestThrottlingManager: assert set(manager._scope_managers) == {"a.example", "b.example"} def test_scope_limit_disabled(self): - manager = _manager({"THROTTLING_SCOPE_LIMIT": 0}) + manager = _manager({"THROTTLER_SCOPE_LIMIT": 0}) for i in range(5): scope = manager._get_scope_manager(f"{i}.example") scope.record_sent(now=0.0) @@ -455,7 +455,7 @@ class TestThrottlingManager: assert len(manager._scope_managers) == 5 -class TestThrottlingScopeManager: +class TestThrottlerScopeManager: def test_no_delay_by_default(self): scope = _scope_manager() scope.record_sent(now=0.0) @@ -588,10 +588,10 @@ class TestThrottlingScopeManager: assert scope._concurrency == 8 def test_no_scope_concurrency_limit_when_zero(self): - # THROTTLING_SCOPE_CONCURRENCY governs scopes that are neither a domain + # THROTTLER_SCOPE_CONCURRENCY governs scopes that are neither a domain # nor an IP (here a bare "custom" group name). scope = _scope_manager( - settings={"THROTTLING_SCOPE_CONCURRENCY": 0}, config={"id": "custom"} + settings={"THROTTLER_SCOPE_CONCURRENCY": 0}, config={"id": "custom"} ) assert scope._concurrency is None for _ in range(100): @@ -702,8 +702,8 @@ class TestThrottlingScopeManager: assert scope._consumed == pytest.approx(7.0) -class TestThrottlingManagerReadiness: - """The synchronous readiness API used by a throttling-aware scheduler.""" +class TestThrottlerReadiness: + """The synchronous readiness API used by a throttler-aware scheduler.""" @coroutine_test async def test_is_ready_unconstrained_scope(self): @@ -725,7 +725,7 @@ class TestThrottlingManagerReadiness: async def test_reserve_blocks_delay_scope(self): manager = _manager( { - "THROTTLING_SCOPES": {"example.com": {"delay": 100.0}}, + "THROTTLER_SCOPES": {"example.com": {"delay": 100.0}}, "RANDOMIZE_DOWNLOAD_DELAY": False, } ) @@ -740,28 +740,28 @@ class TestThrottlingManagerReadiness: assert manager.get_time_until_ready(second) == pytest.approx(100.0, abs=1.0) @coroutine_test - async def test_throttling_delay_blocks_until_deadline(self): - manager = _manager({"THROTTLING_DEBUG": True}) - request = Request("http://example.com/a", meta={"throttling_delay": 100.0}) + async def test_delay_blocks_until_deadline(self): + manager = _manager({"THROTTLER_DEBUG": True}) + request = Request("http://example.com/a", meta={"delay": 100.0}) await manager.get_scopes(request) # The per-request delay holds back the request even though its scope is # otherwise unconstrained. assert manager.is_ready(request) is False assert manager.get_time_until_ready(request) == pytest.approx(100.0, abs=1.0) # The deadline is computed once and reused by later polls. - deadline = request.meta["_throttling_delay_deadline"] + deadline = request.meta["_throttler_delay_deadline"] assert manager.is_ready(request) is False - assert request.meta["_throttling_delay_deadline"] == deadline + assert request.meta["_throttler_delay_deadline"] == deadline @coroutine_test - async def test_throttling_delay_not_reapplied_once_consumed(self): + async def test_delay_not_reapplied_once_consumed(self): # A request whose delay was already honored (e.g. promoted out of a - # throttling-aware queue's holding area, or restored on resume) is + # throttler-aware queue's holding area, or restored on resume) is # ready, so it cannot re-block its scope set on a stale deadline. manager = _manager() request = Request( "http://example.com/a", - meta={"throttling_delay": 100.0, "_throttling_delayed": True}, + meta={"delay": 100.0, "_throttler_delayed": True}, ) await manager.get_scopes(request) assert manager.is_ready(request) is True @@ -771,16 +771,14 @@ class TestThrottlingManagerReadiness: async def test_get_request_delay(self): manager = _manager() assert manager.get_request_delay( - Request("http://example.com/a", meta={"throttling_delay": 100.0}) + Request("http://example.com/a", meta={"delay": 100.0}) ) == pytest.approx(100.0, abs=1.0) # A request without a per-request delay is not held individually. assert manager.get_request_delay(Request("http://example.com/b")) == 0.0 @coroutine_test async def test_delay_scope(self): - manager = _manager( - {"THROTTLING_DEBUG": True, "RANDOMIZE_DOWNLOAD_DELAY": False} - ) + manager = _manager({"THROTTLER_DEBUG": True, "RANDOMIZE_DOWNLOAD_DELAY": False}) request = Request("http://example.com/a") await manager.get_scopes(request) assert manager.is_ready(request) is True @@ -804,7 +802,7 @@ class TestThrottlingManagerReadiness: @coroutine_test async def test_reserve_blocks_on_concurrency(self): - manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}}) + manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}}) first = Request("http://example.com/1") second = Request("http://example.com/2") await manager.get_scopes(first) @@ -820,7 +818,7 @@ class TestThrottlingManagerReadiness: async def test_acquire_noop_when_reserved(self): manager = _manager( { - "THROTTLING_SCOPES": {"example.com": {"delay": 100.0}}, + "THROTTLER_SCOPES": {"example.com": {"delay": 100.0}}, "RANDOMIZE_DOWNLOAD_DELAY": False, } ) @@ -835,7 +833,7 @@ class TestThrottlingManagerReadiness: @coroutine_test async def test_get_scope_load(self): - manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 4}}}) + manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 4}}}) assert manager.get_scope_load("example.com") == 0.0 request = Request("http://example.com/1") await manager.get_scopes(request) @@ -937,11 +935,11 @@ class TestScopeHelpers: update_scope_backoff({"a": 1.0}, "a", delay=1.0) -class TestThrottlingManagerEdges: +class TestThrottlerEdges: @coroutine_test async def test_acquire_without_scopes(self): manager = _manager() - request = Request("http://example.com/a", meta={"throttling_scopes": []}) + request = Request("http://example.com/a", meta={"throttler_scopes": []}) # No scopes resolve, so acquire() returns without reserving anything. await manager.acquire(request) assert request not in manager._reserved @@ -956,9 +954,9 @@ class TestThrottlingManagerEdges: async def test_acquire_logs_and_waits_for_delay(self): manager = _manager( { - "THROTTLING_SCOPES": {"example.com": {"delay": 0.02}}, + "THROTTLER_SCOPES": {"example.com": {"delay": 0.02}}, "RANDOMIZE_DOWNLOAD_DELAY": False, - "THROTTLING_DEBUG": True, + "THROTTLER_DEBUG": True, } ) r1 = Request("http://example.com/1") @@ -974,8 +972,8 @@ class TestThrottlingManagerEdges: manager = _manager( { - "THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}, - "THROTTLING_DEBUG": True, + "THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}, + "THROTTLER_DEBUG": True, } ) r1 = Request("http://example.com/1") @@ -988,10 +986,10 @@ class TestThrottlingManagerEdges: @coroutine_test async def test_delay_request(self): - manager = _manager({"THROTTLING_DEBUG": True}) - request = Request("http://example.com/a", meta={"throttling_delay": 0.01}) + manager = _manager({"THROTTLER_DEBUG": True}) + request = Request("http://example.com/a", meta={"delay": 0.01}) await manager._delay_request(request) - assert request.meta["_throttling_delayed"] is True + assert request.meta["_throttler_delayed"] is True # A second call is a no-op (the request was already delayed). await manager._delay_request(request) @@ -1000,9 +998,9 @@ class TestThrottlingManagerEdges: # Same as above but with debug logging off, so the delay is applied # without emitting the debug message. manager = _manager() - request = Request("http://example.com/a", meta={"throttling_delay": 0.01}) + request = Request("http://example.com/a", meta={"delay": 0.01}) await manager._delay_request(request) - assert request.meta["_throttling_delayed"] is True + assert request.meta["_throttler_delayed"] is True @coroutine_test async def test_wait_for_slot_discards_unfired_events(self): @@ -1028,14 +1026,14 @@ class TestThrottlingManagerEdges: assert manager._get_scope_manager("example.com")._backoff_level == 1 def test_apply_backoff_debug_logging(self, caplog): - manager = _manager({"THROTTLING_DEBUG": True}) - with caplog.at_level(logging.DEBUG, logger="scrapy.throttling"): + manager = _manager({"THROTTLER_DEBUG": True}) + with caplog.at_level(logging.DEBUG, logger="scrapy.throttler"): manager._apply_backoff("example.com") assert "Backoff for scope" in caplog.text assert manager._get_scope_manager("example.com")._backoff_level == 1 def test_on_robots_parsed_disabled(self): - manager = _manager({"THROTTLING_ROBOTSTXT_OBEY": False}) + manager = _manager({"THROTTLER_ROBOTSTXT_OBEY": False}) # With obeying disabled, the handler returns without touching any scope. manager._on_robots_parsed(_FakeRobotParser(5.0), Request("http://example.com")) assert not manager._scope_managers @@ -1044,21 +1042,21 @@ class TestThrottlingManagerEdges: manager = _manager( { "ROBOTSTXT_OBEY": True, - "THROTTLING_SCOPES": {"example.com": {"delay": 0.5}}, + "THROTTLER_SCOPES": {"example.com": {"delay": 0.5}}, } ) - with caplog.at_level(logging.WARNING, logger="scrapy.throttling"): + with caplog.at_level(logging.WARNING, logger="scrapy.throttler"): manager.apply_robots_crawl_delay("example.com", 3.0) assert "Crawl-delay" in caplog.text def test_apply_robots_crawl_delay_debug_logging(self, caplog): - manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLING_DEBUG": True}) - with caplog.at_level(logging.DEBUG, logger="scrapy.throttling"): + manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLER_DEBUG": True}) + with caplog.at_level(logging.DEBUG, logger="scrapy.throttler"): manager.apply_robots_crawl_delay("example.com", 3.0) assert "Crawl-delay" in caplog.text def test_maybe_evict_disabled(self): - manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 0}) + manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 0}) scope = manager._get_scope_manager("example.com") scope.record_sent(now=0.0) scope.record_done(now=0.0) @@ -1067,7 +1065,7 @@ class TestThrottlingManagerEdges: assert "example.com" in manager._scope_managers -class TestThrottlingScopeManagerEdges: +class TestThrottlerScopeManagerEdges: def test_jitter_as_range(self): scope = _scope_manager( config={"id": "x", "backoff": {"jitter": [0.0, 0.0]}, "min_delay": 1.0} @@ -1134,7 +1132,7 @@ class TestThrottlingScopeManagerEdges: assert scope.is_idle(now=0.0, max_idle=1.0) is True -class TestThrottlingIntegration: +class TestThrottlerIntegration: @coroutine_test async def test_backoff_recorded_on_429(self, mockserver): crawler = get_crawler(SimpleSpider, {"RETRY_ENABLED": False}) @@ -1144,21 +1142,21 @@ class TestThrottlingIntegration: throttler = crawler.throttler assert throttler is not None managers = throttler._scope_managers - assert managers, "no throttling scope was created" + assert managers, "no throttler scope was created" assert any(manager._backoff_level >= 1 for manager in managers.values()) @coroutine_test async def test_backoff_recorded_on_download_error(self, mockserver): crawler = get_crawler(SimpleSpider, {"RETRY_ENABLED": False}) # A dropped connection raises a DownloadFailedError, which the engine - # routes through the throttling manager before re-raising. + # routes through the throttler before re-raising. await crawler.crawl_async( mockserver.url("/drop?abort=1"), mockserver=mockserver ) throttler = crawler.throttler assert throttler is not None managers = throttler._scope_managers - assert managers, "no throttling scope was created" + assert managers, "no throttler scope was created" assert any(manager._backoff_level >= 1 for manager in managers.values()) @coroutine_test