RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

throttling → throttler

This commit is contained in:
Adrian Chaves 2026-07-03 16:18:36 +02:00
parent 8fcb544765
commit f0cc534737
23 changed files with 564 additions and 578 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/intro/tutorial.rst
View File

@ -141,12 +141,12 @@ parallel.
``quotes.toscrape.com`` is a sandbox meant for scraping practice, so we can
safely crawl it faster. Open ``tutorial/settings.py`` and add a
:setting:`THROTTLING_SCOPES` entry that raises the concurrency and lowers the
:setting:`THROTTLER_SCOPES` entry that raises the concurrency and lowers the
delay for that domain only:
.. code-block:: python
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"quotes.toscrape.com": {"concurrency": 16, "delay": 0.1},
}

2
docs/topics/broad-crawls.rst
View File

@ -48,7 +48,7 @@ Increase concurrency
Concurrency is the number of requests that are processed in parallel. There is
a global limit (:setting:`CONCURRENT_REQUESTS`) and an additional per-scope
(per-domain by default) limit (:setting:`THROTTLING_SCOPE_CONCURRENCY`).
(per-domain by default) limit (:setting:`THROTTLER_SCOPE_CONCURRENCY`).
The default global concurrency limit in Scrapy is not suitable for crawling
many different domains in parallel, so you will want to increase it. How much

4
docs/topics/components.rst
View File

@ -24,8 +24,8 @@ That includes the classes that you may assign to the following settings:
- :setting:`SCHEDULER_START_DISK_QUEUE`
- :setting:`SCHEDULER_START_MEMORY_QUEUE`
- :setting:`SPIDER_MIDDLEWARES`
- :setting:`THROTTLING_MANAGER`
- :setting:`THROTTLING_SCOPE_MANAGER`
- :setting:`THROTTLER`
- :setting:`THROTTLER_SCOPE_MANAGER`
- :setting:`TWISTED_DNS_RESOLVER`
Third-party Scrapy components may also let you define additional Scrapy

4
docs/topics/request-response.rst
View File

@ -733,8 +733,8 @@ Those are:
* :reqmeta:`redirect_reasons`
* :reqmeta:`redirect_urls`
* :reqmeta:`referrer_policy`
* :reqmeta:`throttling_delay`
* :reqmeta:`throttling_scopes`
* :reqmeta:`delay`
* :reqmeta:`throttler_scopes`
* :reqmeta:`verbatim_url`
.. reqmeta:: bindaddress

319
docs/topics/throttling.rst
View File

@ -26,12 +26,12 @@ throttling limits, as do ``toscrape.com`` and ``books.toscrape.com``.
The main throttling :ref:`settings <topics-settings>` are:
- .. setting:: THROTTLING_SCOPE_CONCURRENCY
- .. setting:: THROTTLER_SCOPE_CONCURRENCY
:setting:`THROTTLING_SCOPE_CONCURRENCY` (default: ``1``)
:setting:`THROTTLER_SCOPE_CONCURRENCY` (default: ``1``)
Default maximum number of simultaneous requests per :ref:`throttling scope
<throttling-scopes>`. Requests are grouped by domain by default, so this is
Default maximum number of simultaneous requests per :ref:`throttler scope
<throttler-scopes>`. Requests are grouped by domain by default, so this is
the maximum number of simultaneous requests per domain.
It defines a number of “slots” per scope. Each slot can send 1 request at a
@ -51,29 +51,29 @@ The main throttling :ref:`settings <topics-settings>` are:
When configuring these settings, note that:
- :setting:`CONCURRENT_REQUESTS` caps :setting:`THROTTLING_SCOPE_CONCURRENCY`.
- :setting:`CONCURRENT_REQUESTS` caps :setting:`THROTTLER_SCOPE_CONCURRENCY`.
- If ``DOWNLOAD_DELAY`` ≥ response time, concurrency is effectively ``1``,
because the next request to the domain is not sent until the delay elapses,
by which time the previous response has already arrived.
.. [1] You can :ref:`customize <throttling-scopes>` how requests are grouped
.. [1] You can :ref:`customize <throttler-scopes>` how requests are grouped
for throttling, but domain-based throttling works well in most cases. For
more complex domain grouping strategies, see
:ref:`alternative-domain-throttling`.
.. setting:: THROTTLING_SCOPES
.. setting:: THROTTLER_SCOPES
.. _per-domain-throttling:
Per-domain throttling
=====================
The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling
The :setting:`THROTTLER_SCOPES` setting allows you to customize throttling
behavior for specific domains [1]_.
It is a dict that maps scope names to
:class:`~scrapy.throttling.ThrottlingScopeConfig` dicts. It is empty by default.
:class:`~scrapy.throttler.ThrottlerScopeConfig` dicts. It is empty by default.
:command:`startproject` scaffolds a commented-out example entry, so that you can
uncomment and edit it to crawl domains you own (or that are meant for scraping)
@ -82,7 +82,7 @@ other domains:
.. code-block:: python
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"example.com": {"concurrency": 16, "delay": 0.1},
}
@ -133,7 +133,7 @@ See :ref:`throttling-settings` for :setting:`BACKOFF_EXCEPTIONS`,
How backoff works
-----------------
Every :ref:`throttling scope <throttling-scopes>` keeps a current delay that
Every :ref:`throttler scope <throttler-scopes>` keeps a current delay that
starts at its configured ``"delay"`` (:setting:`DOWNLOAD_DELAY` by default).
A **backoff trigger** is a response whose status code is in
@ -181,7 +181,7 @@ Backoff triggers are detected by the
Any component can also trigger backoff programmatically for arbitrary scopes —
e.g. based on the response body of a specific site — through
:meth:`crawler.throttler.back_off()
<scrapy.throttling.ThrottlingManagerProtocol.back_off>`.
<scrapy.throttler.ThrottlerProtocol.back_off>`.
.. _per-scope-backoff:
@ -189,13 +189,13 @@ Per-scope backoff configuration
-------------------------------
The global ``BACKOFF_*`` settings can be overridden per scope with the
``"backoff"`` key of a :setting:`THROTTLING_SCOPES` entry, an instance of
:class:`~scrapy.throttling.BackoffConfig`:
``"backoff"`` key of a :setting:`THROTTLER_SCOPES` entry, an instance of
:class:`~scrapy.throttler.BackoffConfig`:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"example.com": {
"backoff": {
"http_codes": [429, 503],
@ -242,17 +242,17 @@ robots.txt
is a non-standard ``robots.txt`` directive that indicates a number of seconds
to wait between requests.
.. setting:: THROTTLING_ROBOTSTXT_OBEY
.. setting:: THROTTLING_ROBOTSTXT_MAX_DELAY
.. setting:: THROTTLER_ROBOTSTXT_OBEY
.. setting:: THROTTLER_ROBOTSTXT_MAX_DELAY
If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLING_ROBOTSTXT_OBEY` are
If :setting:`ROBOTSTXT_OBEY` and :setting:`THROTTLER_ROBOTSTXT_OBEY` are
``True`` (default), valid ``Crawl-Delay`` directives override
:setting:`THROTTLING_SCOPE_CONCURRENCY` and :setting:`DOWNLOAD_DELAY`.
:setting:`THROTTLER_SCOPE_CONCURRENCY` and :setting:`DOWNLOAD_DELAY`.
Concurrency is set to ``1`` and the delay is raised to at least the
``Crawl-Delay`` value (a larger configured delay is kept), capped at
:setting:`THROTTLING_ROBOTSTXT_MAX_DELAY` (default: ``60.0``).
:setting:`THROTTLER_ROBOTSTXT_MAX_DELAY` (default: ``60.0``).
If :setting:`THROTTLING_SCOPES` defines a different concurrency or delay, it
If :setting:`THROTTLER_SCOPES` defines a different concurrency or delay, it
will be respected, but a warning will be logged about the discrepancy with
``Crawl-Delay``. Set ``ignore_robots_txt`` to ``True`` to silence this warning.
@ -261,9 +261,9 @@ will be respected, but a warning will be logged about the discrepancy with
Delaying a scope programmatically
=================================
You can delay a :ref:`throttling scope <throttling-scopes>` on demand through
You can delay a :ref:`throttler scope <throttler-scopes>` on demand through
:meth:`crawler.throttler.delay_scope()
<scrapy.throttling.ThrottlingManagerProtocol.delay_scope>`:
<scrapy.throttler.ThrottlerProtocol.delay_scope>`:
.. skip: next
@ -315,9 +315,9 @@ For example, you might want to throttle API endpoints differently than web
pages on the same domain, group requests by content type (images vs HTML), or
apply different throttling based on request priority.
.. reqmeta:: throttling_scopes
.. reqmeta:: throttler_scopes
Use the ``throttling_scopes`` request metadata to assign requests to custom
Use the ``throttler_scopes`` request metadata to assign requests to custom
throttling groups:
.. invisible-code-block: python
@ -326,51 +326,51 @@ throttling groups:
.. code-block:: python
Request("https://api.example/", meta={"throttling_scopes": "api"})
Request("https://api.example/", meta={"throttler_scopes": "api"})
You can also assign multiple throttling groups to a single request:
.. code-block:: python
Request("https://api.example/users", meta={"throttling_scopes": {"api", "users"}})
Request("https://api.example/users", meta={"throttler_scopes": {"api", "users"}})
You can then use the :setting:`THROTTLING_SCOPES` setting to customize
You can then use the :setting:`THROTTLER_SCOPES` setting to customize
throttling for such requests:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"api": {"concurrency": 2},
"users": {"delay": 5.0},
}
.. note:: These custom throttling groups persist through redirects. For
redirect-aware throttling assignment, see :ref:`custom-throttling-scopes`.
redirect-aware throttling assignment, see :ref:`custom-throttler-scopes`.
.. reqmeta:: throttling_delay
.. reqmeta:: delay
Delaying a single request
-------------------------
To hold a single request for a fixed number of seconds before it is sent,
regardless of its scopes, set the ``throttling_delay`` request metadata key:
regardless of its scopes, set the ``delay`` request metadata key:
.. code-block:: python
Request("https://example.com/slow", meta={"throttling_delay": 5.0})
Request("https://example.com/slow", meta={"delay": 5.0})
The delay is applied once, the first time the request reaches the throttling
gate.
``throttling_delay`` defines only the *earliest* time the request may be sent,
``delay`` defines only the *earliest* time the request may be sent,
not the exact time: once the delay elapses, the request still competes with
every other pending request for its scopes. If you want it sent **as soon as**
its delay elapses, give it a higher :attr:`~scrapy.Request.priority` too:
.. code-block:: python
Request("https://example.com/slow", meta={"throttling_delay": 5.0}, priority=1)
Request("https://example.com/slow", meta={"delay": 5.0}, priority=1)
Without a higher priority, a backlog of requests ahead of it in a FIFO queue
could keep it waiting well past the configured delay; a higher priority puts it
@ -391,33 +391,33 @@ request normally without letting its outcome trigger :ref:`backoff <backoff>`:
Request("https://example.com/login", meta={"dont_throttle": True})
.. _throttling-scopes:
.. _throttler-scopes:
Throttling scopes
=================
Throttler scopes
================
Throttling scopes represent aspects of requests that can be throttled
Throttler scopes represent aspects of requests that can be throttled
independently.
..
For future reference, the “throttling scope” name was taken from
For future reference, the “throttler scope” name was taken from
https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html#section-1.4-4.4
.. _custom-throttling-scopes:
.. _custom-throttler-scopes:
Customizing throttling scopes
Customizing throttler scopes
------------------------------
There are 2 ways to customize throttling scopes.
There are 2 ways to customize throttler scopes.
To **configure existing scopes**, use the :setting:`THROTTLING_SCOPES` setting.
To **configure existing scopes**, use the :setting:`THROTTLER_SCOPES` setting.
Its keys are scope names and its values are
:class:`~scrapy.throttling.ThrottlingScopeConfig` dicts, which accept the
:class:`~scrapy.throttler.ThrottlerScopeConfig` dicts, which accept the
following keys:
``concurrency`` (:class:`int`)
Maximum number of concurrent requests for the scope. Defaults to
:setting:`THROTTLING_SCOPE_CONCURRENCY`.
:setting:`THROTTLER_SCOPE_CONCURRENCY`.
``delay`` (:class:`float`)
Minimum seconds between requests for the scope. Defaults to
@ -430,152 +430,152 @@ following keys:
a ``[low, high]`` pair multiplies the delay by ``1 + uniform(low, high)``.
``quota`` (:class:`float`)
Maximum :ref:`quota <throttling-quotas>` consumed per ``window``.
Maximum :ref:`quota <throttler-quotas>` consumed per ``window``.
``window`` (:class:`float`)
Quota window in seconds. Defaults to :setting:`THROTTLING_WINDOW`.
Quota window in seconds. Defaults to :setting:`THROTTLER_WINDOW`.
``backoff`` (:class:`~scrapy.throttling.BackoffConfig`)
``backoff`` (:class:`~scrapy.throttler.BackoffConfig`)
Per-scope :ref:`backoff overrides <per-scope-backoff>`.
``manager`` (:class:`str` or :class:`type`)
Import path or class of a :ref:`custom scope manager
<custom-throttling-scope-managers>` for the scope.
<custom-throttler-scope-managers>` for the scope.
``ignore_robots_txt`` (:class:`bool`)
Silences the warning logged when this configuration is more aggressive than
a :ref:`robots.txt Crawl-delay <crawl-delay>`.
.. setting:: THROTTLING_MANAGER
.. setting:: THROTTLER
To **change how scopes are assigned** (or anything beyond per-scope settings),
set :setting:`THROTTLING_MANAGER` (default:
:class:`~scrapy.throttling.ThrottlingManager`) to a :ref:`component
set :setting:`THROTTLER` (default:
:class:`~scrapy.throttler.Throttler`) to a :ref:`component
<topics-components>` that implements the
:class:`~scrapy.throttling.ThrottlingManagerProtocol` protocol (or its import
:class:`~scrapy.throttler.ThrottlerProtocol` protocol (or its import
path as a string):
.. code-block:: python
:caption: ``settings.py``
THROTTLING_MANAGER = "myproject.throttling.MyThrottlingManager"
THROTTLER = "myproject.throttling.MyThrottler"
.. _multiple-throttling-scopes:
.. _multiple-throttler-scopes:
Handling of multiple throttling scopes
Handling of multiple throttler scopes
--------------------------------------
When a request has multiple throttling scopes, it is not sent until all of its
throttling scopes allow it.
When a request has multiple throttler scopes, it is not sent until all of its
throttler scopes allow it.
.. _throttling-quotas:
.. _throttler-quotas:
Throttling quotas
-----------------
Throttler quotas
----------------
When different requests can consume different amounts of a throttling scope,
you can express this using **throttling quotas**.
When different requests can consume different amounts of a throttler scope,
you can express this using **throttler quotas**.
.. setting:: THROTTLING_WINDOW
.. setting:: THROTTLER_WINDOW
Use the :setting:`THROTTLING_WINDOW` setting (default: ``60.0``) or the ``"window"``
key in the :setting:`THROTTLING_SCOPES` setting to define the time window after
which throttling quotas are reset.
Use the :setting:`THROTTLER_WINDOW` setting (default: ``60.0``) or the ``"window"``
key in the :setting:`THROTTLER_SCOPES` setting to define the time window after
which throttler quotas are reset.
Then use the :setting:`THROTTLING_SCOPES` setting to define the throttling
quotas for each throttling scope:
Then use the :setting:`THROTTLER_SCOPES` setting to define the throttling
quotas for each throttler scope:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"api.toscrape.com": {
"quota": 500.0,
},
}
Then, in the :reqmeta:`throttling_scopes` request metadata key or in the return
value of the :meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scopes`
method, define a :class:`dict` where keys are throttling scopes and values are
Then, in the :reqmeta:`throttler_scopes` request metadata key or in the return
value of the :meth:`~scrapy.throttler.ThrottlerProtocol.get_scopes`
method, define a :class:`dict` where keys are throttler scopes and values are
:class:`float` values that indicate the expected quota consumption (it does not
need to be exact).
.. _custom-throttling-scope-managers:
.. _custom-throttler-scope-managers:
Customizing throttling scope managers
Customizing throttler scope managers
-------------------------------------
.. setting:: THROTTLING_SCOPE_MANAGER
.. setting:: THROTTLER_SCOPE_MANAGER
The :setting:`THROTTLING_SCOPE_MANAGER` setting (default:
:class:`~scrapy.throttling.ThrottlingScopeManager`) is a :ref:`component
The :setting:`THROTTLER_SCOPE_MANAGER` setting (default:
:class:`~scrapy.throttler.ThrottlerScopeManager`) is a :ref:`component
<topics-components>` that implements the
:class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` (or its import path
:class:`~scrapy.throttler.ThrottlerScopeManagerProtocol` (or its import path
as a string):
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPE_MANAGER = "myproject.throttling.MyThrottlingScopeManager"
THROTTLER_SCOPE_MANAGER = "myproject.throttling.MyThrottlerScopeManager"
For each throttling scope, an instance of this class is created to manage any
For each throttler scope, an instance of this class is created to manage any
gradual :ref:`backoff <backoff>` required at run time.
You can implement your own throttling scope manager if you wish to change the
You can implement your own throttler scope manager if you wish to change the
backoff behavior beyond what settings allow.
You can also define a custom throttling scope manager for a specific throttling
scope by setting the ``"manager"`` key in the :setting:`THROTTLING_SCOPES`
You can also define a custom throttler scope manager for a specific throttling
scope by setting the ``"manager"`` key in the :setting:`THROTTLER_SCOPES`
setting:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"api.toscrape.com": {
"manager": "myproject.throttling.MyThrottlingScopeManager",
"manager": "myproject.throttling.MyThrottlerScopeManager",
},
}
Most custom scope managers subclass the default
:class:`~scrapy.throttling.ThrottlingScopeManager` and override only the methods
:class:`~scrapy.throttler.ThrottlerScopeManager` and override only the methods
whose behavior they want to change; implementing the
:class:`~scrapy.throttling.ThrottlingScopeManagerProtocol` from scratch is also
:class:`~scrapy.throttler.ThrottlerScopeManagerProtocol` from scratch is also
supported. For example, this manager disables exponential :ref:`backoff
<backoff>`, so a scope relies solely on its configured delay and quota:
.. code-block:: python
:caption: ``myproject/throttling.py``
from scrapy.throttling import ThrottlingScopeManager
from scrapy.throttler import ThrottlerScopeManager
class FixedWindowScopeManager(ThrottlingScopeManager):
class FixedWindowScopeManager(ThrottlerScopeManager):
def record_backoff(self, *args, **kwargs):
pass # never back off
.. _throttling-aware-scheduler:
.. _throttler-aware-scheduler:
Throttling-aware scheduling
===========================
By default, throttling is enforced at the engine, where a request waiting on
its :ref:`throttling scopes <throttling-scopes>` holds a concurrency slot. In a
its :ref:`throttler scopes <throttler-scopes>` holds a concurrency slot. In a
crawl that mixes heavily-throttled scopes with unthrottled ones, this can let
throttled requests starve unthrottled ones that could be sent right away
(**head-of-line blocking**; Scrapy logs a warning the first time throttled
requests start consuming the global concurrency budget while they wait).
:class:`~scrapy.core.scheduler.ThrottlingAwareScheduler` avoids this. To enable
:class:`~scrapy.core.scheduler.ThrottlerAwareScheduler` avoids this. To enable
it:
.. code-block:: python
:caption: ``settings.py``
SCHEDULER = "scrapy.core.scheduler.ThrottlingAwareScheduler"
SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.ThrottlingAwarePriorityQueue"
SCHEDULER = "scrapy.core.scheduler.ThrottlerAwareScheduler"
SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.ThrottlerAwarePriorityQueue"
.. autoclass:: scrapy.core.scheduler.ThrottlingAwareScheduler
.. autoclass:: scrapy.core.scheduler.ThrottlerAwareScheduler
.. _throttling-examples:
@ -587,14 +587,14 @@ Examples
Alternative domain throttling
-----------------------------
If you are not happy with the :ref:`default throttling scope behavior
If you are not happy with the :ref:`default throttler scope behavior
<basic-throttling>` with regards to domains and subdomains, you can change it.
Alternative approaches include:
- Using the **highest-level registrable domain** as the throttling scope,
- Using the **highest-level registrable domain** as the throttler scope,
e.g. https://books.toscrape.com and https://toscrape.com both get a
``toscrape.com`` throttling scope.
``toscrape.com`` throttler scope.
This allows to apply the same throttling settings to all subdomains of a
registrable domain.
@ -605,24 +605,24 @@ Alternative approaches include:
:caption: ``settings.py``
import tldextract
from scrapy.throttling import ThrottlingManager, scope_cache
from scrapy.throttler import Throttler, scope_cache
from scrapy.utils.httpobj import urlparse_cached
class MyThrottlingManager(ThrottlingManager):
class MyThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
extracted = tldextract.extract(request.url)
return extracted.registered_domain or urlparse_cached(request).netloc
THROTTLING_MANAGER = MyThrottlingManager
THROTTLER = MyThrottler
- Using **multiple throttling scopes per request**, one per registrable
- Using **multiple throttler scopes per request**, one per registrable
domain and for every higher-level subdomain, e.g.
https://books.toscrape.com and https://toscrape.com both get a
``toscrape.com`` throttling scope, but https://books.toscrape.com also
gets a ``books.toscrape.com`` throttling scope.
``toscrape.com`` throttler scope, but https://books.toscrape.com also
gets a ``books.toscrape.com`` throttler scope.
This allows to apply the same throttling settings to all subdomains of a
registrable domain, but also allows applying further restrictions on each
@ -634,11 +634,11 @@ Alternative approaches include:
:caption: ``settings.py``
import tldextract
from scrapy.throttling import ThrottlingManager, scope_cache
from scrapy.throttler import Throttler, scope_cache
from scrapy.utils.httpobj import urlparse_cached
class MyThrottlingManager(ThrottlingManager):
class MyThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
extracted = tldextract.extract(request.url)
@ -648,8 +648,8 @@ Alternative approaches include:
return {extracted.registered_domain, extracted.fqdn}
THROTTLING_MANAGER = MyThrottlingManager
THROTTLING_SCOPES = {
THROTTLER = MyThrottler
THROTTLER_SCOPES = {
"toscrape.com": {"concurrency": 32},
"books.toscrape.com": {"concurrency": 24},
"quotes.toscrape.com": {"concurrency": 16},
@ -669,16 +669,16 @@ To apply different throttling settings to different endpoints of the same
domain and not enforce any common throttling, effectively treating them as
different domains:
- Implement a :ref:`throttling manager <custom-throttling-scopes>` that sets
endpoint-specific throttling scopes for that domain:
- Implement a :ref:`throttler <custom-throttler-scopes>` that sets
endpoint-specific throttler scopes for that domain:
.. code-block:: python
from scrapy.throttling import ThrottlingManager, scope_cache
from scrapy.throttler import Throttler, scope_cache
from scrapy.utils.httpobj import urlparse_cached
class MyThrottlingManager(ThrottlingManager):
class MyThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
parsed_url = urlparse_cached(request)
@ -686,13 +686,13 @@ different domains:
return await super().get_scopes(request)
return f"{parsed_url.netloc}{parsed_url.path}"
- Use the :setting:`THROTTLING_SCOPES` setting to set different throttling
- Use the :setting:`THROTTLER_SCOPES` setting to set different throttling
settings per endpoint:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"api.toscrape.com/fast-endpoint": {"concurrency": 1000, "delay": 0.08},
"api.toscrape.com/slow-endpoint": {"delay": 5.0},
}
@ -707,35 +707,35 @@ Imagine you are sending requests to a web scraping API, e.g. to avoid bans.
Unless that API provides a Scrapy plugin to make it easier to use, you may want
to:
- Use the :setting:`THROTTLING_SCOPES` setting to increase concurrency for
- Use the :setting:`THROTTLER_SCOPES` setting to increase concurrency for
API requests. For example:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"api.toscrape.com": {"concurrency": 1000, "delay": 0.08},
}
- Implement a :ref:`throttling manager <custom-throttling-scopes>` that:
- Implement a :ref:`throttler <custom-throttler-scopes>` that:
- Adds a throttling scope for the URL being scraped.
- Adds a throttler scope for the URL being scraped.
For example, if you request
``https://api.toscrape.com/?url=https://example.com``, by default it
will get a ``api.toscrape.com`` throttling scope, but it should also
get the ``example.com`` throttling scope:
will get a ``api.toscrape.com`` throttler scope, but it should also
get the ``example.com`` throttler scope:
.. code-block:: python
from urllib.parse import urlparse
from scrapy.throttling import add_scope, ThrottlingManager, scope_cache
from scrapy.throttler import add_scope, Throttler, scope_cache
from scrapy.utils.httpobj import urlparse_cached
from w3lib.url import url_query_parameter
class MyThrottlingManager(ThrottlingManager):
class MyThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
scopes = await super().get_scopes(request)
@ -756,7 +756,7 @@ to:
.. code-block:: python
from scrapy.throttling import iter_scopes
from scrapy.throttler import iter_scopes
from scrapy.utils.httpobj import urlparse_cached
@ -795,21 +795,21 @@ Cost-capped throttling
Imagine you are using an API that charges different requests differently, e.g.
based on the features used, and you want to limit how much you spend per time
window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas
<throttling-quotas>` for that:
window (:setting:`THROTTLER_WINDOW`). You can use :ref:`throttler quotas
<throttler-quotas>` for that:
- Implement a :ref:`throttling manager <custom-throttling-scopes>` that:
- Implement a :ref:`throttler <custom-throttler-scopes>` that:
- Sets a ``cost`` throttling scope on each request to some estimation
- Sets a ``cost`` throttler scope on each request to some estimation
based e.g. on request URL parameters:
.. code-block:: python
from scrapy.utils.httpobj import urlparse_cached
from scrapy.throttling import ThrottlingManager, scope_cache
from scrapy.throttler import Throttler, scope_cache
class MyThrottlingManager(ThrottlingManager):
class MyThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
scopes = await super().get_scopes(request)
@ -840,13 +840,13 @@ window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas
self.throttler.reconcile_quota("cost", consumed=actual - estimated)
return response
- Use the :setting:`THROTTLING_SCOPES` setting to set a maximum cost per time
- Use the :setting:`THROTTLER_SCOPES` setting to set a maximum cost per time
window:
.. code-block:: python
:caption: ``settings.py``
THROTTLING_SCOPES = {
THROTTLER_SCOPES = {
"cost": {"quota": 100.0},
}
@ -858,24 +858,24 @@ window (:setting:`THROTTLING_WINDOW`). You can use :ref:`throttling quotas
Per-IP concurrency limiting
---------------------------
A concurrency limit keyed by IP is just a throttling scope whose id is the
A concurrency limit keyed by IP is just a throttler scope whose id is the
request's IP, with a ``concurrency`` limit. A request then carries two scopes,
its domain and its IP, and is only sent when **both** allow it (see
:ref:`multiple-throttling-scopes`).
:ref:`multiple-throttler-scopes`).
- Implement a :ref:`throttling manager <custom-throttling-scopes>` that adds
- Implement a :ref:`throttler <custom-throttler-scopes>` that adds
the request's IP as a second scope:
.. code-block:: python
import socket
from scrapy.throttling import ThrottlingManager, add_scope, scope_cache
from scrapy.throttler import Throttler, add_scope, scope_cache
from scrapy.utils.asyncio import run_in_thread
from scrapy.utils.httpobj import urlparse_cached
class IPThrottlingManager(ThrottlingManager):
class IPThrottler(Throttler):
@scope_cache
async def get_scopes(self, request):
scopes = await super().get_scopes(request)
@ -887,6 +887,7 @@ its domain and its IP, and is only sent when **both** allow it (see
Additional settings
===================
- .. setting:: BACKOFF_ENABLED
:setting:`BACKOFF_ENABLED` (default: ``True``)
@ -933,7 +934,7 @@ Additional settings
:setting:`BACKOFF_WINDOW` (default: ``60.0``)
Time window, in seconds, used by :ref:`backoff <backoff>`. A
:ref:`throttling scope <throttling-scopes>` must go this many seconds
:ref:`throttler scope <throttler-scopes>` must go this many seconds
without a new backoff
trigger (an HTTP error code from :setting:`BACKOFF_HTTP_CODES` or an
exception from :setting:`BACKOFF_EXCEPTIONS`) before its delay decreases
@ -954,18 +955,18 @@ Additional settings
If ``True``, ``0.5`` (i.e. ±50%) is used as the randomization factor. If
``False``, no randomization is applied.
- .. setting:: THROTTLING_DEBUG
- .. setting:: THROTTLER_DEBUG
:setting:`THROTTLING_DEBUG` (default: ``False``)
:setting:`THROTTLER_DEBUG` (default: ``False``)
Whether to log :ref:`throttling <throttling>` decisions (per-scope delays,
backoff steps and recoveries) for debugging.
- .. setting:: THROTTLING_SCOPE_LIMIT
- .. setting:: THROTTLER_SCOPE_LIMIT
:setting:`THROTTLING_SCOPE_LIMIT` (default: ``100000``)
:setting:`THROTTLER_SCOPE_LIMIT` (default: ``100000``)
Maximum number of :ref:`throttling scope <throttling-scopes>` states kept
Maximum number of :ref:`throttler scope <throttler-scopes>` states kept
in memory at once, to bound memory usage on broad crawls that touch a large
number of scopes (e.g. domains).
@ -975,15 +976,15 @@ Additional settings
evicted, so the limit may be temporarily exceeded if that many scopes are
busy at once. Set to ``0`` to disable the limit.
This complements :setting:`THROTTLING_SCOPE_MAX_IDLE`, which evicts scopes
This complements :setting:`THROTTLER_SCOPE_MAX_IDLE`, which evicts scopes
by inactivity time rather than by count.
- .. setting:: THROTTLING_SCOPE_MAX_IDLE
- .. setting:: THROTTLER_SCOPE_MAX_IDLE
:setting:`THROTTLING_SCOPE_MAX_IDLE` (default: ``3600.0``)
:setting:`THROTTLER_SCOPE_MAX_IDLE` (default: ``3600.0``)
Seconds of inactivity after which the state of a :ref:`throttling scope
<throttling-scopes>` is evicted from memory to bound memory usage on
Seconds of inactivity after which the state of a :ref:`throttler scope
<throttler-scopes>` is evicted from memory to bound memory usage on
long-running crawls. Set to ``0`` to never evict. Scopes in active backoff
are never evicted.
@ -992,24 +993,24 @@ Additional settings
API
===
.. autoclass:: scrapy.throttling.ThrottlingManagerProtocol
.. autoclass:: scrapy.throttler.ThrottlerProtocol
:members:
:member-order: bysource
.. autoclass:: scrapy.throttling.ThrottlingManager
.. autoclass:: scrapy.throttler.Throttler
.. autoclass:: scrapy.throttling.ThrottlingScopeManagerProtocol
.. autoclass:: scrapy.throttler.ThrottlerScopeManagerProtocol
:members:
:member-order: bysource
.. autoclass:: scrapy.throttling.ThrottlingScopeManager
.. autoclass:: scrapy.throttler.ThrottlerScopeManager
.. autoclass:: scrapy.pqueues.ThrottlingAwarePriorityQueue
.. autoclass:: scrapy.pqueues.ThrottlerAwarePriorityQueue
.. autoclass:: scrapy.throttling.ThrottlingScopeConfig
.. autoclass:: scrapy.throttler.ThrottlerScopeConfig
.. autoclass:: scrapy.throttling.BackoffConfig
.. autoclass:: scrapy.throttler.BackoffConfig
.. autofunction:: scrapy.throttling.scope_cache
.. autofunction:: scrapy.throttling.add_scope
.. autofunction:: scrapy.throttling.iter_scopes
.. autofunction:: scrapy.throttler.scope_cache
.. autofunction:: scrapy.throttler.add_scope
.. autofunction:: scrapy.throttler.iter_scopes

19
scrapy/core/downloader/__init__.py
View File

@ -28,10 +28,7 @@ if TYPE_CHECKING:
from scrapy.http import Response
from scrapy.settings import BaseSettings
from scrapy.signalmanager import SignalManager
from scrapy.throttling import (
ThrottlingManagerProtocol,
ThrottlingScopeManagerProtocol,
)
from scrapy.throttler import ThrottlerProtocol, ThrottlerScopeManagerProtocol
from scrapy.utils.asyncio import CallLaterResult
@ -92,7 +89,7 @@ class _DeprecatedSlotView:
self,
downloader: Downloader,
key: str,
scope: ThrottlingScopeManagerProtocol,
scope: ThrottlerScopeManagerProtocol,
) -> None:
self._downloader = downloader
self._key = key
@ -114,10 +111,10 @@ class _DeprecatedSlotView:
if r.meta.get(Downloader.DOWNLOAD_SLOT) == self._key
}
# This deprecated view reads throttling scope state from private attributes
# This deprecated view reads throttler scope state from private attributes
# of the default scope manager rather than through the scope manager
# protocol: these are read-only compatibility accessors, so keeping them off
# the protocol avoids forcing custom THROTTLING_SCOPE_MANAGER implementations
# the protocol avoids forcing custom THROTTLER_SCOPE_MANAGER implementations
# to provide members that only exist to feed this shim. A custom manager that
# lacks the attribute simply falls back to the historical default.
@property
@ -168,9 +165,7 @@ class _DeprecatedSlotsView(Mapping[str, _DeprecatedSlotView]):
__slots__ = ("_downloader", "_throttler")
def __init__(
self, downloader: Downloader, throttler: ThrottlingManagerProtocol
) -> None:
def __init__(self, downloader: Downloader, throttler: ThrottlerProtocol) -> None:
self._downloader = downloader
self._throttler = throttler
@ -216,7 +211,7 @@ class Downloader:
)
if self.per_slot_settings:
warnings.warn(
"The DOWNLOAD_SLOTS setting is deprecated. Use THROTTLING_SCOPES for "
"The DOWNLOAD_SLOTS setting is deprecated. Use THROTTLER_SCOPES for "
"per-domain configuration instead.",
category=ScrapyDeprecationWarning,
stacklevel=2,
@ -264,7 +259,7 @@ class Downloader:
@property
def slots(self) -> _DeprecatedSlotsView:
warnings.warn(
"Downloader.slots is deprecated. Use the throttling manager API instead.",
"Downloader.slots is deprecated. Use the throttler API instead.",
category=ScrapyDeprecationWarning,
stacklevel=2,
)

6
scrapy/core/downloader/handlers/_base_http.py
View File

@ -17,7 +17,7 @@ class BaseHttpDownloadHandler(BaseDownloadHandler, ABC):
def _max_per_host_concurrency(settings: BaseSettings) -> int:
"""Highest per-host concurrency the throttler may admit: the per-domain
limit, the default ``other``-scope limit, and any explicit
:setting:`THROTTLING_SCOPES` concurrency.
:setting:`THROTTLER_SCOPES` concurrency.
Since :setting:`CONCURRENT_REQUESTS` caps the total number of requests in
flight, no host can ever exceed it, so it is also the upper bound of the
@ -26,11 +26,11 @@ class BaseHttpDownloadHandler(BaseDownloadHandler, ABC):
global_concurrency = settings.getint("CONCURRENT_REQUESTS")
candidates = [
settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN"),
settings.getint("THROTTLING_SCOPE_CONCURRENCY"),
settings.getint("THROTTLER_SCOPE_CONCURRENCY"),
]
candidates += [
int(scope["concurrency"])
for scope in settings.getdict("THROTTLING_SCOPES").values()
for scope in settings.getdict("THROTTLER_SCOPES").values()
if "concurrency" in scope
]
return min(max(candidates), global_concurrency)

42
scrapy/core/engine.py
View File

@ -131,9 +131,9 @@ class ExecutionEngine:
self._start_request_processing_awaitable: (
asyncio.Future[None] | Deferred[None] | None
) = None
# Requests currently held by the throttling manager, waiting for their
# Requests currently held by the throttler, waiting for their
# scopes to allow them through to the downloader.
self._throttling_waiting: set[Request] = set()
self._throttler_waiting: set[Request] = set()
# Number of in-flight asynchronous enqueue operations (see
# ``_enqueue_request_async``), so the spider is not considered idle
# while a request is still on its way into the scheduler.
@ -149,7 +149,7 @@ class ExecutionEngine:
# Whether the scheduler exposes enqueue_request_async, cached once per
# crawl in ``open_spider_async``; checked for every scheduled request.
self._scheduler_enqueues_async: bool = False
self._throttling_backout_warned: bool = False
self._throttler_backout_warned: bool = False
downloader_cls: type[Downloader] = load_object(self.settings["DOWNLOADER"])
try:
self.scheduler_cls: type[BaseScheduler] = self._get_scheduler_class(
@ -412,40 +412,40 @@ class ExecutionEngine:
or bool(self._slot.closing)
or self.downloader.needs_backout()
or self.scraper.slot.needs_backout()
or self._throttling_needs_backout()
or self._throttler_needs_backout()
)
def _throttling_needs_backout(self) -> bool:
"""Apply backpressure while requests are held by the throttling manager.
def _throttler_needs_backout(self) -> bool:
"""Apply backpressure while requests are held by the throttler.
Requests waiting on throttling do not occupy a downloader concurrency
slot, so without this check the engine would keep draining the
scheduler into pending throttling waits. Count them together with the
in-flight requests against the global concurrency limit.
"""
if not self._throttling_waiting:
if not self._throttler_waiting:
return False
if (
len(self._throttling_waiting) + len(self.downloader.active)
len(self._throttler_waiting) + len(self.downloader.active)
< self.downloader.total_concurrency
):
return False
self._maybe_warn_throttling_backout()
self._maybe_warn_throttler_backout()
return True
def _maybe_warn_throttling_backout(self) -> None:
if self._throttling_backout_warned:
def _maybe_warn_throttler_backout(self) -> None:
if self._throttler_backout_warned:
return
# A throttling-aware scheduler holds time-blocked requests itself instead
# of letting them pile up in _throttling_waiting and back-pressure the
# A throttler-aware scheduler holds time-blocked requests itself instead
# of letting them pile up in _throttler_waiting and back-pressure the
# engine here, so the warning does not apply when one is in use.
if self._get_next_request_delay is not None:
return
self._throttling_backout_warned = True
self._throttler_backout_warned = True
logger.warning(
"Throttling is holding requests back and they are now consuming the "
"global concurrency budget while they wait for a free slot. Consider "
"switching to scrapy.core.scheduler.ThrottlingAwareScheduler, which "
"switching to scrapy.core.scheduler.ThrottlerAwareScheduler, which "
"holds throttled requests without consuming concurrency slots.",
extra={"spider": self.spider},
)
@ -523,7 +523,7 @@ class ExecutionEngine:
return False
if self.downloader.active: # downloader has pending requests
return False
if self._throttling_waiting: # requests held by the throttling manager
if self._throttler_waiting: # requests held by the throttler
return False
if self._scheduling: # requests still on their way into the scheduler
return False
@ -612,18 +612,18 @@ class ExecutionEngine:
return response_or_request
@inlineCallbacks
def _acquire_throttling(
def _acquire_throttler(
self, request: Request
) -> Generator[Deferred[Any], Any, None]:
"""Wait at the throttling gate before *request* is sent, tracking it as
held meanwhile."""
self._throttling_waiting.add(request)
self._throttler_waiting.add(request)
throttler = self.crawler.throttler
assert throttler is not None
try:
yield deferred_from_coro(throttler.acquire(request))
finally:
self._throttling_waiting.discard(request)
self._throttler_waiting.discard(request)
@inlineCallbacks
def _download(
@ -634,12 +634,12 @@ class ExecutionEngine:
throttler = self.crawler.throttler
assert throttler is not None
# A throttling-aware scheduler reserves the request (a concurrency slot)
# A throttler-aware scheduler reserves the request (a concurrency slot)
# before handing it here, so everything past this point runs inside the
# try/finally that releases it, even if add_request() were to raise.
try:
self._slot.add_request(request)
yield self._acquire_throttling(request)
yield self._acquire_throttler(request)
result: Response | Request
if self._downloader_fetch_needs_spider:
result = yield self.downloader.fetch(request, self.spider)

28
scrapy/core/scheduler.py
View File

@ -10,7 +10,7 @@ from typing import TYPE_CHECKING, Any, cast
from twisted.internet.defer import Deferred # noqa: TC002
from scrapy.spiders import Spider # noqa: TC001
from scrapy.throttling import iter_scopes
from scrapy.throttler import iter_scopes
from scrapy.utils.job import job_dir
from scrapy.utils.misc import build_from_crawler, load_object
@ -26,7 +26,7 @@ if TYPE_CHECKING:
from scrapy.http.request import Request
from scrapy.pqueues import ScrapyPriorityQueue
from scrapy.statscollectors import StatsCollector
from scrapy.throttling import ThrottlingManagerProtocol
from scrapy.throttler import ThrottlerProtocol
logger = logging.getLogger(__name__)
@ -79,7 +79,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
Asynchronous counterpart of :meth:`enqueue_request`, following the same
return contract. When a scheduler defines it, the engine awaits it
*instead of* calling :meth:`enqueue_request`. Define it when enqueuing a
request needs to ``await`` (e.g. to resolve throttling scopes
request needs to ``await`` (e.g. to resolve throttler scopes
asynchronously).
.. method:: get_next_request_delay()
@ -238,7 +238,7 @@ class Scheduler(BaseScheduler):
-------------------------
While pending requests are below the configured values of
:setting:`CONCURRENT_REQUESTS` or :setting:`THROTTLING_SCOPE_CONCURRENCY`,
:setting:`CONCURRENT_REQUESTS` or :setting:`THROTTLER_SCOPE_CONCURRENCY`,
those requests are sent concurrently.
As a result, the first few requests of a crawl may not follow the desired
@ -410,7 +410,7 @@ class Scheduler(BaseScheduler):
Extra positional arguments are forwarded to the underlying queue
``push`` calls; subclasses that store requests under additional keys
(e.g. a throttling scope set) pass them through here.
(e.g. a throttler scope set) pass them through here.
"""
assert self.stats is not None
if self._dqpush(request, *push_args):
@ -535,14 +535,14 @@ class Scheduler(BaseScheduler):
json.dump(state, f)
class ThrottlingAwareScheduler(Scheduler):
class ThrottlerAwareScheduler(Scheduler):
"""A :setting:`SCHEDULER` that only ever hands the engine requests that
their :ref:`throttling scopes <throttling-scopes>` allow to be sent **right
their :ref:`throttler scopes <throttler-scopes>` allow to be sent **right
now**.
The default scheduler hands requests to the engine as concurrency allows
and lets them wait at the throttling gate
(:meth:`~scrapy.throttling.ThrottlingManagerProtocol.acquire`) while holding
(:meth:`~scrapy.throttler.ThrottlerProtocol.acquire`) while holding
a concurrency slot. When a crawl mixes heavily-throttled scopes with
unthrottled ones, enough throttled requests waiting on a clock can fill
:setting:`CONCURRENT_REQUESTS` and starve unthrottled requests that could be
@ -551,7 +551,7 @@ class ThrottlingAwareScheduler(Scheduler):
(beyond what is needed to track each distinct scope set) and cannot starve
other scopes.
It also honors the per-request :reqmeta:`throttling_delay`, holding an
It also honors the per-request :reqmeta:`delay`, holding an
individual request back without blocking others that share its scopes, which
the default scheduler cannot do.
@ -560,7 +560,7 @@ class ThrottlingAwareScheduler(Scheduler):
broken by preferring the least-busy scopes.
It requires :setting:`SCHEDULER_PRIORITY_QUEUE` to be set to
:class:`~scrapy.pqueues.ThrottlingAwarePriorityQueue` (or a compatible
:class:`~scrapy.pqueues.ThrottlerAwarePriorityQueue` (or a compatible
subclass).
"""
@ -569,18 +569,18 @@ class ThrottlingAwareScheduler(Scheduler):
if not hasattr(self.mqs, "get_next_request_delay"):
raise ValueError(
f"{type(self).__name__} requires SCHEDULER_PRIORITY_QUEUE to be "
f"set to a throttling-aware priority queue such as "
f"scrapy.pqueues.ThrottlingAwarePriorityQueue, but the "
f"set to a throttler-aware priority queue such as "
f"scrapy.pqueues.ThrottlerAwarePriorityQueue, but the "
f"configured one ({type(self.mqs).__name__}) is not."
)
assert self.crawler is not None
assert self.crawler.throttler is not None
self._throttler: ThrottlingManagerProtocol = self.crawler.throttler
self._throttler: ThrottlerProtocol = self.crawler.throttler
return result
def enqueue_request(self, request: Request) -> bool:
raise RuntimeError(
"ThrottlingAwareScheduler requires the asynchronous enqueue path; "
"ThrottlerAwareScheduler requires the asynchronous enqueue path; "
"enqueue_request_async() is used by the engine instead of "
"enqueue_request()."
)

16
scrapy/crawler.py
View File

@ -46,7 +46,7 @@ if TYPE_CHECKING:
from scrapy.logformatter import LogFormatter
from scrapy.statscollectors import StatsCollector
from scrapy.throttling import ThrottlingManagerProtocol
from scrapy.throttler import ThrottlerProtocol
from scrapy.utils.request import RequestFingerprinterProtocol
@ -84,13 +84,13 @@ class Crawler:
self.stats: StatsCollector | None = None
self.logformatter: LogFormatter | None = None
self.request_fingerprinter: RequestFingerprinterProtocol | None = None
self.throttler: ThrottlingManagerProtocol | None = None
"""The throttling manager of this crawler, an instance of
:setting:`THROTTLING_MANAGER`.
self.throttler: ThrottlerProtocol | None = None
"""The throttler of this crawler, an instance of
:setting:`THROTTLER`.
It is ``None`` until the crawl starts. Components can use it to inspect
or drive :ref:`throttling <throttling>` at run time, e.g. through
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.delay_scope`."""
:meth:`~scrapy.throttler.ThrottlerProtocol.delay_scope`."""
self.spider: Spider | None = None
self.engine: ExecutionEngine | None = None
@ -106,7 +106,7 @@ class Crawler:
self.addons.load_settings(self.settings)
self._apply_deprecated_spider_attr("download_delay", "DOWNLOAD_DELAY")
self._apply_deprecated_spider_attr(
"max_concurrent_requests", "THROTTLING_SCOPE_CONCURRENCY"
"max_concurrent_requests", "THROTTLER_SCOPE_CONCURRENCY"
)
self.stats = load_object(self.settings["STATS_CLASS"])(self)
@ -118,7 +118,7 @@ class Crawler:
self,
)
self.throttler = build_from_crawler(
load_object(self.settings["THROTTLING_MANAGER"]),
load_object(self.settings["THROTTLER"]),
self,
)
@ -185,7 +185,7 @@ class Crawler:
return
warnings.warn(
f"The {attr!r} spider attribute is deprecated. Use the {setting} "
f"setting or THROTTLING_SCOPES instead.",
f"setting or THROTTLER_SCOPES instead.",
category=ScrapyDeprecationWarning,
stacklevel=3,
)

12
scrapy/downloadermiddlewares/backoff.py
View File

@ -4,7 +4,7 @@ import logging
from typing import TYPE_CHECKING
from scrapy.exceptions import NotConfigured
from scrapy.throttling import iter_scopes
from scrapy.throttler import iter_scopes
from scrapy.utils._headers import _parse_ratelimit_reset, _parse_retry_after
from scrapy.utils.decorators import _warn_spider_arg
from scrapy.utils.misc import _load_objects
@ -16,7 +16,7 @@ if TYPE_CHECKING:
import scrapy
from scrapy.crawler import Crawler
from scrapy.http import Request, Response
from scrapy.throttling import ThrottlingManagerProtocol
from scrapy.throttler import ThrottlerProtocol
logger = logging.getLogger(__name__)
@ -28,9 +28,9 @@ class BackoffMiddleware:
It observes every response and download exception and, for those matching
:setting:`BACKOFF_HTTP_CODES` or :setting:`BACKOFF_EXCEPTIONS` (globally or
per :setting:`THROTTLING_SCOPES` scope), tells the :ref:`throttling manager
per :setting:`THROTTLER_SCOPES` scope), tells the :ref:`throttler
<throttling>` to back off the request's scopes through its
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.back_off` API.
:meth:`~scrapy.throttler.ThrottlerProtocol.back_off` API.
It is enabled by default; set :setting:`BACKOFF_ENABLED` to ``False`` to
disable it without removing it from :setting:`DOWNLOADER_MIDDLEWARES`.
@ -46,7 +46,7 @@ class BackoffMiddleware:
if not crawler.settings.getbool("BACKOFF_ENABLED"):
raise NotConfigured
assert crawler.throttler is not None
self._throttler: ThrottlingManagerProtocol = crawler.throttler
self._throttler: ThrottlerProtocol = crawler.throttler
settings = crawler.settings
self._http_codes: set[int] = {
int(code) for code in settings.getlist("BACKOFF_HTTP_CODES")
@ -61,7 +61,7 @@ class BackoffMiddleware:
# pre-filter to skip outcomes that no scope backs off on.
self._any_http_codes: set[int] = set(self._http_codes)
self._any_exceptions: tuple[type[BaseException], ...] = self._exceptions
for scope_id, scope_config in settings.getdict("THROTTLING_SCOPES").items():
for scope_id, scope_config in settings.getdict("THROTTLER_SCOPES").items():
backoff = scope_config.get("backoff") or {}
if "http_codes" in backoff:
codes = {int(code) for code in backoff["http_codes"]}

8
scrapy/downloadermiddlewares/redirect.py
View File

@ -184,7 +184,7 @@ class BaseRedirectMiddleware:
"""Delay the redirect when *response* carries a ``Retry-After`` header.
The delay is capped at :setting:`REDIRECT_MAX_DELAY` and applied through
the :reqmeta:`throttling_delay` request metadata key, which holds back
the :reqmeta:`delay` request metadata key, which holds back
this request only (without counting as a :ref:`backoff <backoff>`
trigger for its scopes). :setting:`REDIRECT_MAX_DELAY` set to ``0``
disables it.
@ -194,11 +194,11 @@ class BaseRedirectMiddleware:
retry_after = _parse_retry_after(response)
if retry_after is None:
return
redirect_request.meta["throttling_delay"] = min(retry_after, self.max_delay)
redirect_request.meta["delay"] = min(retry_after, self.max_delay)
# This is a fresh request that may inherit an already-honored delay from
# its source request's meta; clear that state so the new delay applies.
redirect_request.meta.pop("_throttling_delayed", None)
redirect_request.meta.pop("_throttling_delay_deadline", None)
redirect_request.meta.pop("_throttler_delayed", None)
redirect_request.meta.pop("_throttler_delay_deadline", None)
def _redirect_request_using_get(
self, request: Request, response: Response, redirect_url: str

8
scrapy/extensions/throttle.py
View File

@ -14,7 +14,7 @@ if TYPE_CHECKING:
from scrapy.crawler import Crawler
from scrapy.http import Response
from scrapy.throttling import ThrottlingManagerProtocol
from scrapy.throttler import ThrottlerProtocol
logger = logging.getLogger(__name__)
@ -28,7 +28,7 @@ class AutoThrottle:
warn(
"You have set the AUTOTHROTTLE_ENABLED setting to True, however "
"the AutoThrottle extension is deprecated; use throttling and "
"the AutoThrottle extension is deprecated; use throttler and "
"backoff settings instead: "
"https://docs.scrapy.org/en/latest/topics/throttling.html",
ScrapyDeprecationWarning,
@ -99,9 +99,7 @@ class AutoThrottle:
extra={"spider": spider},
)
def _scope_delay(
self, throttler: ThrottlingManagerProtocol, scope_id: str
) -> float:
def _scope_delay(self, throttler: ThrottlerProtocol, scope_id: str) -> float:
"""Return the current delay of *scope_id*, applying AUTOTHROTTLE_START_DELAY
the first time the scope is seen."""
delay = throttler.get_scope_delay(scope_id)

68
scrapy/pqueues.py
View File

@ -17,7 +17,7 @@ if TYPE_CHECKING:
from scrapy import Request
from scrapy.crawler import Crawler
from scrapy.throttling import ScopeID, ThrottlingManagerProtocol
from scrapy.throttler import ScopeID, ThrottlerProtocol
logger = logging.getLogger(__name__)
@ -329,7 +329,7 @@ class DownloaderAwarePriorityQueue:
)
assert crawler.throttler is not None
self._throttler: ThrottlingManagerProtocol = crawler.throttler
self._throttler: ThrottlerProtocol = crawler.throttler
self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls
self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls
self.key: str = key
@ -441,10 +441,10 @@ def _scope_set_from_key(key: str) -> frozenset[ScopeID]:
return frozenset(json.loads(key))
class ThrottlingAwarePriorityQueue:
class ThrottlerAwarePriorityQueue:
"""Priority queue that only ever pops a request that can be sent right now
based on its :ref:`throttling scope set <throttling-scopes>` and
per-request :reqmeta:`throttling_delay`.
based on its :ref:`throttler scope set <throttler-scopes>` and
per-request :reqmeta:`delay`.
The downstream queue class must support ``peek``.
@ -460,6 +460,22 @@ class ThrottlingAwarePriorityQueue:
persistence directory, and inside it this class creates a subdirectory per
scope set, named from a path-safe, order-independent encoding of its scope
ids.
For example, a request whose scope set is ``{"example.com",
"cost:group-1"}`` is stored under a subdirectory derived in two steps:
#. The scope ids are sorted and JSON-encoded into an order-independent key
(so ``{"example.com", "cost:group-1"}`` and ``{"cost:group-1",
"example.com"}`` map to the same one)::
["cost:group-1", "example.com"]
#. That key is made path-safe: every character outside ``[A-Za-z0-9-._]``
becomes ``_`` (here the ``:`` and the JSON quotes, brackets and spaces;
the ``-`` and ``.`` are kept), and an MD5 suffix disambiguates keys that
collapse to the same path::
__cost_group-1____example.com__-fc6ba2aff8f421bf981b662d77739902
"""
@classmethod
@ -491,7 +507,7 @@ class ThrottlingAwarePriorityQueue:
):
if slot_startprios and not isinstance(slot_startprios, dict):
raise ValueError(
"ThrottlingAwarePriorityQueue accepts ``slot_startprios`` as a "
"ThrottlerAwarePriorityQueue accepts ``slot_startprios`` as a "
f"dict; {slot_startprios.__class__!r} instance is passed. Most "
"likely, it means the state is created by an incompatible "
"priority queue. Only a crawl started with the same priority "
@ -499,7 +515,7 @@ class ThrottlingAwarePriorityQueue:
)
assert crawler.throttler is not None
self._throttler: ThrottlingManagerProtocol = crawler.throttler
self._throttler: ThrottlerProtocol = crawler.throttler
self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls
self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls
self.key: str = key
@ -509,15 +525,26 @@ class ThrottlingAwarePriorityQueue:
if slot_startprios:
for set_key, startprios in slot_startprios.items():
scope_set = _scope_set_from_key(set_key)
self.pqueues[scope_set] = self.pqfactory(scope_set, startprios)
self.pqueues[scope_set] = self._pqfactory(scope_set, startprios)
# Min-heap of (deadline, seq, scope_set, request) for requests held
# back by a per-request throttling_delay; seq keeps ordering stable and
# avoids comparing requests when deadlines tie.
# Requests held back by their own per-request delay wait
# here instead of in their scope-set queue, so a not-yet-due request
# never sits at a queue head and blocks the other requests that share
# its scopes (the head-of-line blocking this scheduler exists to
# avoid). Once the delay elapses, _promote_ready() moves the request
# into its scope-set queue, where it competes normally.
#
# This is a min-heap of (deadline, seq, scope_set, request). heapq
# needs a total order and always compares entries to keep its invariant;
# seq (a monotonic counter) makes the (deadline, seq) prefix totally
# ordered, so a deadline tie is broken there and the scope_set (a
# frozenset, whose < is only the partial subset order) and the request
# (no ordering at all) are never compared. The order among tied
# deadlines is irrelevant beyond being deterministic.
self._delayed: list[tuple[float, int, frozenset[ScopeID], Request]] = []
self._delayed_seq: int = 0
def pqfactory(
def _pqfactory(
self, scope_set: frozenset[ScopeID], startprios: Iterable[int] = ()
) -> ScrapyPriorityQueue:
return ScrapyPriorityQueue(
@ -542,7 +569,7 @@ class ThrottlingAwarePriorityQueue:
def _push_to_queue(self, request: Request, scope_set: frozenset[ScopeID]) -> None:
if scope_set not in self.pqueues:
self.pqueues[scope_set] = self.pqfactory(scope_set)
self.pqueues[scope_set] = self._pqfactory(scope_set)
self.pqueues[scope_set].push(request)
def _promote_ready(self, now: float) -> None:
@ -559,7 +586,7 @@ class ThrottlingAwarePriorityQueue:
# mark it consumed: the request must not be delayed again, and on resume
# it must not re-block its scope set on a stale, no-longer-meaningful
# deadline.
request.meta["_throttling_delayed"] = True
request.meta["_throttler_delayed"] = True
try:
self._push_to_queue(request, scope_set)
except ValueError as e:
@ -586,7 +613,7 @@ class ThrottlingAwarePriorityQueue:
Among the sendable queues (those whose scope set can be sent right now),
the one whose head has the highest request priority is chosen; ties are
broken by ascending load (the maximum
:meth:`~scrapy.throttling.ThrottlingManagerProtocol.get_scope_load` over the
:meth:`~scrapy.throttler.ThrottlerProtocol.get_scope_load` over the
scopes of the queue), i.e. by preferring the least-busy scopes.
"""
self._promote_ready(time.monotonic())
@ -618,12 +645,6 @@ class ThrottlingAwarePriorityQueue:
del self.pqueues[scope_set]
return request
def peek(self) -> Request | None:
selected = self._select()
if selected is None:
return None
return selected[1].peek()
def get_next_request_delay(self) -> float | None:
now = time.monotonic()
self._promote_ready(now)
@ -639,7 +660,7 @@ class ThrottlingAwarePriorityQueue:
continue
if delay is None or head_delay < delay:
delay = head_delay
# A request held back only by its own throttling_delay is not in any
# A request held back only by its own delay is not in any
# scope-set queue, so factor in when the earliest one is due.
if self._delayed:
next_delayed = max(0.0, self._delayed[0][0] - now)
@ -662,6 +683,3 @@ class ThrottlingAwarePriorityQueue:
def __len__(self) -> int:
queued = sum(len(x) for x in self.pqueues.values()) if self.pqueues else 0
return queued + len(self._delayed)
def __contains__(self, scope_set: frozenset[ScopeID]) -> bool:
return scope_set in self.pqueues

40
scrapy/settings/default_settings.py
View File

@ -217,16 +217,16 @@ __all__ = [
"TELNETCONSOLE_PORT",
"TELNETCONSOLE_USERNAME",
"TEMPLATES_DIR",
"THROTTLING_DEBUG",
"THROTTLING_MANAGER",
"THROTTLING_ROBOTSTXT_MAX_DELAY",
"THROTTLING_ROBOTSTXT_OBEY",
"THROTTLING_SCOPES",
"THROTTLING_SCOPE_CONCURRENCY",
"THROTTLING_SCOPE_LIMIT",
"THROTTLING_SCOPE_MANAGER",
"THROTTLING_SCOPE_MAX_IDLE",
"THROTTLING_WINDOW",
"THROTTLER",
"THROTTLER_DEBUG",
"THROTTLER_ROBOTSTXT_MAX_DELAY",
"THROTTLER_ROBOTSTXT_OBEY",
"THROTTLER_SCOPES",
"THROTTLER_SCOPE_CONCURRENCY",
"THROTTLER_SCOPE_LIMIT",
"THROTTLER_SCOPE_MANAGER",
"THROTTLER_SCOPE_MAX_IDLE",
"THROTTLER_WINDOW",
"TWISTED_DNS_RESOLVER",
"TWISTED_REACTOR",
"TWISTED_REACTOR_ENABLED",
@ -607,16 +607,16 @@ TELNETCONSOLE_PASSWORD = None
TEMPLATES_DIR = str((Path(__file__).parent / ".." / "templates").resolve())
THROTTLING_MANAGER = "scrapy.throttling.ThrottlingManager"
THROTTLING_SCOPE_MANAGER = "scrapy.throttling.ThrottlingScopeManager"
THROTTLING_SCOPES = {}
THROTTLING_WINDOW = 60.0
THROTTLING_ROBOTSTXT_OBEY = True
THROTTLING_ROBOTSTXT_MAX_DELAY = 60.0
THROTTLING_SCOPE_CONCURRENCY = 1
THROTTLING_SCOPE_LIMIT = 100000
THROTTLING_SCOPE_MAX_IDLE = 3600.0
THROTTLING_DEBUG = False
THROTTLER = "scrapy.throttler.Throttler"
THROTTLER_SCOPE_MANAGER = "scrapy.throttler.ThrottlerScopeManager"
THROTTLER_SCOPES = {}
THROTTLER_WINDOW = 60.0
THROTTLER_ROBOTSTXT_OBEY = True
THROTTLER_ROBOTSTXT_MAX_DELAY = 60.0
THROTTLER_SCOPE_CONCURRENCY = 1
THROTTLER_SCOPE_LIMIT = 100000
THROTTLER_SCOPE_MAX_IDLE = 3600.0
THROTTLER_DEBUG = False
TWISTED_DNS_RESOLVER = "scrapy.resolver.CachingThreadedResolver"

3
scrapy/templates/project/module/settings.py.tmpl
View File

@ -21,9 +21,8 @@ ADDONS = {}
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Concurrency and throttling settings
#CONCURRENT_REQUESTS = 16
THROTTLING_SCOPE_CONCURRENCY = 1
THROTTLER_SCOPE_CONCURRENCY = 1
DOWNLOAD_DELAY = 1
# Disable cookies (enabled by default)

212
scrapy/throttling.py → scrapy/throttler.py
View File

@ -33,7 +33,7 @@ logger = logging.getLogger(__name__)
class BackoffConfig(TypedDict, total=False):
"""Per-scope override of the backoff settings.
Used as the value of the ``"backoff"`` key of :class:`ThrottlingScopeConfig`
Used as the value of the ``"backoff"`` key of :class:`ThrottlerScopeConfig`
entries. Any key left out falls back to the corresponding global
``BACKOFF_*`` setting.
"""
@ -46,8 +46,8 @@ class BackoffConfig(TypedDict, total=False):
jitter: float | list[float]
class ThrottlingScopeConfig(TypedDict, total=False):
"""Accepted keys of :setting:`THROTTLING_SCOPES` entries.
class ThrottlerScopeConfig(TypedDict, total=False):
"""Accepted keys of :setting:`THROTTLER_SCOPES` entries.
Every key is optional; missing keys fall back to the matching global
setting (e.g. ``delay`` falls back to :setting:`DOWNLOAD_DELAY`).
@ -66,7 +66,7 @@ class ThrottlingScopeConfig(TypedDict, total=False):
window: float
manager: str | type
"""Import path or class of a custom :setting:`THROTTLING_SCOPE_MANAGER` for
"""Import path or class of a custom :setting:`THROTTLER_SCOPE_MANAGER` for
this scope."""
backoff: BackoffConfig
@ -83,8 +83,8 @@ RequestScopes = None | ScopeID | Iterable[ScopeID] | dict[ScopeID, float | None]
def iter_scopes(scopes: RequestScopes) -> Iterable[ScopeID]:
"""Iterate over the scope IDs of *scopes*, whatever its form.
:class:`~ThrottlingManagerProtocol.get_scopes` (and
:meth:`~ThrottlingManagerProtocol.get_resolved_scopes`) may return a single
:class:`~ThrottlerProtocol.get_scopes` (and
:meth:`~ThrottlerProtocol.get_resolved_scopes`) may return a single
scope ID, an iterable of them, a ``{scope_id: quota}`` mapping, or ``None``;
this helper normalizes any of those into an iterable of scope IDs, e.g. to
react to a request's scopes in a custom middleware.
@ -95,7 +95,7 @@ def iter_scopes(scopes: RequestScopes) -> Iterable[ScopeID]:
def iter_scope_values(scopes: RequestScopes) -> Iterable[tuple[ScopeID, float | None]]:
"""Iterate over *scopes* as ``(scope_id, value)`` pairs.
For dict scopes the value is the expected :ref:`throttling quota
For dict scopes the value is the expected :ref:`throttler quota
<throttling-quotas>` consumption; for every other form the value is
``None``.
"""
@ -120,42 +120,42 @@ def _effective_priority(settings: BaseSettings, name: str) -> int:
def _default_scope_concurrency(settings: BaseSettings) -> int:
"""Return the default concurrency of a throttling scope that does not set
"""Return the default concurrency of a throttler scope that does not set
its own ``concurrency``.
This is :setting:`THROTTLING_SCOPE_CONCURRENCY`, except that the deprecated
This is :setting:`THROTTLER_SCOPE_CONCURRENCY`, except that the deprecated
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting is bridged in when set at a
higher :ref:`priority <populating-settings>`. When neither is set
explicitly, the historical :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` value is
kept for backward compatibility (its default flips to
:setting:`THROTTLING_SCOPE_CONCURRENCY` in a future version; see
:setting:`THROTTLER_SCOPE_CONCURRENCY` in a future version; see
:func:`_warn_on_deprecated_concurrency`).
"""
default_priority = SETTINGS_PRIORITIES["default"]
domain_priority = _effective_priority(settings, "CONCURRENT_REQUESTS_PER_DOMAIN")
scope_priority = _effective_priority(settings, "THROTTLING_SCOPE_CONCURRENCY")
scope_priority = _effective_priority(settings, "THROTTLER_SCOPE_CONCURRENCY")
if domain_priority > scope_priority:
return settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN")
if scope_priority > domain_priority:
return settings.getint("THROTTLING_SCOPE_CONCURRENCY")
return settings.getint("THROTTLER_SCOPE_CONCURRENCY")
# Equal priority: on an explicit (higher-than-default) tie the new setting
# wins; when neither is set (both at "default") keep the historical
# per-domain value so existing behavior is preserved.
if domain_priority <= default_priority:
return settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN")
return settings.getint("THROTTLING_SCOPE_CONCURRENCY")
return settings.getint("THROTTLER_SCOPE_CONCURRENCY")
def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None:
"""Warn about the concurrency settings bridged by
:func:`_default_scope_concurrency`. Call once per crawl (see
:meth:`ThrottlingManager.__init__`).
:meth:`Throttler.__init__`).
When :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is set explicitly, warn that
it is deprecated. When neither concurrency setting is set explicitly, warn
that the effective per-scope concurrency is still the deprecated setting's
value (kept for backward compatibility) and will drop to
:setting:`THROTTLING_SCOPE_CONCURRENCY`'s default once the deprecated
:setting:`THROTTLER_SCOPE_CONCURRENCY`'s default once the deprecated
setting is removed, so users can pin it explicitly."""
default_priority = SETTINGS_PRIORITIES["default"]
domain_set = (
@ -163,12 +163,12 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None:
> default_priority
)
scope_set = (
_effective_priority(settings, "THROTTLING_SCOPE_CONCURRENCY") > default_priority
_effective_priority(settings, "THROTTLER_SCOPE_CONCURRENCY") > default_priority
)
if domain_set:
warnings.warn(
"The CONCURRENT_REQUESTS_PER_DOMAIN setting is deprecated, use "
"THROTTLING_SCOPE_CONCURRENCY instead.",
"THROTTLER_SCOPE_CONCURRENCY instead.",
category=ScrapyDeprecationWarning,
stacklevel=2,
)
@ -178,14 +178,14 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None:
# guarded by test_deprecated_concurrency_defaults_differ rather than at
# run time, so a crawl is never aborted over it.
current = settings.getint("CONCURRENT_REQUESTS_PER_DOMAIN")
future = settings.getint("THROTTLING_SCOPE_CONCURRENCY")
future = settings.getint("THROTTLER_SCOPE_CONCURRENCY")
warnings.warn(
f"The effective per-scope (per-domain) concurrency is {current}, "
f"the default of the deprecated CONCURRENT_REQUESTS_PER_DOMAIN "
f"setting, which is still respected for backward compatibility. "
f"Once CONCURRENT_REQUESTS_PER_DOMAIN is removed, it will drop to "
f"{future}, the default of THROTTLING_SCOPE_CONCURRENCY. Set "
f"THROTTLING_SCOPE_CONCURRENCY explicitly to choose a value and "
f"{future}, the default of THROTTLER_SCOPE_CONCURRENCY. Set "
f"THROTTLER_SCOPE_CONCURRENCY explicitly to choose a value and "
f"silence this warning.",
category=ScrapyDeprecationWarning,
stacklevel=2,
@ -195,7 +195,7 @@ def _warn_on_deprecated_concurrency(settings: BaseSettings) -> None:
def _warn_on_unachievable_concurrency(settings: BaseSettings) -> None:
"""Warn about configured concurrency limits that exceed
:setting:`CONCURRENT_REQUESTS`. Call once per crawl (see
:meth:`ThrottlingManager.__init__`).
:meth:`Throttler.__init__`).
:setting:`CONCURRENT_REQUESTS` caps the total number of requests in flight,
so a per-scope (or per-domain) concurrency limit above it can never be
@ -204,12 +204,12 @@ def _warn_on_unachievable_concurrency(settings: BaseSettings) -> None:
global_concurrency = settings.getint("CONCURRENT_REQUESTS")
offenders: list[str] = [
f"{name}={settings.getint(name)}"
for name in ("CONCURRENT_REQUESTS_PER_DOMAIN", "THROTTLING_SCOPE_CONCURRENCY")
for name in ("CONCURRENT_REQUESTS_PER_DOMAIN", "THROTTLER_SCOPE_CONCURRENCY")
if settings.getint(name) > global_concurrency
]
offenders += [
f"THROTTLING_SCOPES[{scope_id!r}]['concurrency']={config['concurrency']}"
for scope_id, config in settings.getdict("THROTTLING_SCOPES").items()
f"THROTTLER_SCOPES[{scope_id!r}]['concurrency']={config['concurrency']}"
for scope_id, config in settings.getdict("THROTTLER_SCOPES").items()
if config.get("concurrency") is not None
and int(config["concurrency"]) > global_concurrency
]
@ -249,8 +249,8 @@ def add_scope(
dict.
This is a utility function to help extending the output of
:meth:`~ThrottlingManagerProtocol.get_scopes`, e.g. in
:class:`ThrottlingManager` subclasses.
:meth:`~ThrottlerProtocol.get_scopes`, e.g. in
:class:`Throttler` subclasses.
Adding a scope with a *value* fails if it is already present, so an existing
:ref:`quota <throttling-quotas>` is never silently overwritten; adding it
@ -266,21 +266,21 @@ def add_scope(
return result
class ThrottlingManagerProtocol(Protocol):
"""A protocol for :setting:`THROTTLING_MANAGER` :ref:`components
class ThrottlerProtocol(Protocol):
"""A protocol for :setting:`THROTTLER` :ref:`components
<topics-components>`."""
async def get_scopes(self, request: Request) -> RequestScopes:
"""Return the :ref:`throttling scopes <throttling-scopes>` that apply
"""Return the :ref:`throttler scopes <throttling-scopes>` that apply
to *request*.
Return ``None`` if no scopes apply, a string for a single scope, an
iterable of strings for multiple scopes, or a dict with scope names as
keys and :ref:`throttling quotas <throttling-quotas>` as values.
keys and :ref:`throttler quotas <throttling-quotas>` as values.
"""
def get_resolved_scopes(self, request: Request) -> RequestScopes:
"""Return the :ref:`throttling scopes <throttling-scopes>` under which
"""Return the :ref:`throttler scopes <throttling-scopes>` under which
*request* was (or will be) sent, without re-resolving them.
This is the synchronous counterpart of :meth:`get_scopes`: it returns
@ -314,8 +314,8 @@ class ThrottlingManagerProtocol(Protocol):
*and* a concurrency slot is free in every scope.
This is the synchronous, non-blocking counterpart of :meth:`acquire`,
used by a :ref:`throttling-aware scheduler
<throttling-aware-scheduler>` to decide whether a request can be
used by a :ref:`throttler-aware scheduler
<throttler-aware-scheduler>` to decide whether a request can be
dequeued now. It assumes the scopes of *request* have already been
resolved (e.g. by an earlier :meth:`get_scopes` call at enqueue time).
"""
@ -325,7 +325,7 @@ class ThrottlingManagerProtocol(Protocol):
scopes and mark *request* as reserved, so that a later :meth:`acquire`
for it returns immediately without reserving again.
A :ref:`throttling-aware scheduler <throttling-aware-scheduler>` calls
A :ref:`throttler-aware scheduler <throttler-aware-scheduler>` calls
this when it decides to dequeue *request* (after :meth:`is_ready`
returned ``True``). The reservation is released by :meth:`release`.
"""
@ -335,8 +335,8 @@ class ThrottlingManagerProtocol(Protocol):
*request* would be open, or ``None`` if no time-based gate is currently
blocking it (only a concurrency slot could be).
Used by a :ref:`throttling-aware scheduler
<throttling-aware-scheduler>` to schedule a wakeup when all pending
Used by a :ref:`throttler-aware scheduler
<throttler-aware-scheduler>` to schedule a wakeup when all pending
requests are time-blocked.
"""
@ -354,19 +354,19 @@ class ThrottlingManagerProtocol(Protocol):
active sends divided by its concurrency limit (or by the global
:setting:`CONCURRENT_REQUESTS` when the scope has no explicit limit).
Used by a :ref:`throttling-aware scheduler
<throttling-aware-scheduler>` to balance dequeuing across scopes,
Used by a :ref:`throttler-aware scheduler
<throttler-aware-scheduler>` to balance dequeuing across scopes,
preferring the least-loaded ones.
"""
def get_request_delay(self, request: Request, now: float | None = None) -> float:
"""Return how many seconds *request* must still be held individually
because of its :reqmeta:`throttling_delay`, or ``0.0`` if it has none
because of its :reqmeta:`delay`, or ``0.0`` if it has none
or it has already elapsed. The one-time delay is started on the first
call.
Unlike a scope delay, this affects only *request*: a
:ref:`throttling-aware scheduler <throttling-aware-scheduler>` must
:ref:`throttler-aware scheduler <throttler-aware-scheduler>` must
hold the request back on its own, **without** blocking other requests
that share its scopes.
"""
@ -423,7 +423,7 @@ class ThrottlingManagerProtocol(Protocol):
consumed: float | None = None,
remaining: float | None = None,
) -> None:
"""Reconcile the :ref:`throttling quota <throttling-quotas>` of each of
"""Reconcile the :ref:`throttler quota <throttling-quotas>` of each of
*scopes* with an actually *consumed* amount (a delta to add) or a
*remaining* amount (an absolute value), correcting the estimate used
when requests were sent.
@ -444,8 +444,8 @@ class ThrottlingManagerProtocol(Protocol):
directly (e.g. an adaptive-delay extension).
"""
def get_scope_manager(self, scope_id: str) -> ThrottlingScopeManagerProtocol:
"""Return the :class:`ThrottlingScopeManagerProtocol` instance handling
def get_scope_manager(self, scope_id: str) -> ThrottlerScopeManagerProtocol:
"""Return the :class:`ThrottlerScopeManagerProtocol` instance handling
the scope identified by *scope_id*, creating it if necessary."""
@ -456,16 +456,16 @@ _GetScopesMethod = TypeVar(
# Request.meta key under which scope_cache persists the resolved scopes so that
# they survive a request being serialized to and restored from a disk queue.
_RESOLVED_SCOPES_META_KEY = "_throttling_resolved_scopes"
_RESOLVED_SCOPES_META_KEY = "_throttler_resolved_scopes"
def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod:
"""Decorator for :meth:`~ThrottlingManagerProtocol.get_scopes`
"""Decorator for :meth:`~ThrottlerProtocol.get_scopes`
implementations that persists the resolved scopes on ``request.meta``.
The readers of the resolved scopes — the synchronous readiness API of a
:ref:`throttling-aware scheduler <throttling-aware-scheduler>` and
:meth:`~ThrottlingManagerProtocol.get_resolved_scopes` — read this persisted
:ref:`throttler-aware scheduler <throttler-aware-scheduler>` and
:meth:`~ThrottlerProtocol.get_resolved_scopes` — read this persisted
value instead of resolving the scopes again, so they stay cheap and
consistent, and it survives a request being
serialized to and restored from a :ref:`disk queue <topics-jobs>` (which is
@ -483,10 +483,10 @@ def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod:
.. code-block:: python
from scrapy.utils.httpobj import urlparse_cached
from scrapy.throttling import scope_cache
from scrapy.throttler import scope_cache
class MyThrottlingManager:
class MyThrottler:
@scope_cache
async def get_scopes(self, request):
return urlparse_cached(request).hostname or ""
@ -505,8 +505,8 @@ def scope_cache(f: _GetScopesMethod) -> _GetScopesMethod:
return wrapper # type: ignore[return-value]
class ThrottlingManager:
"""The default :setting:`THROTTLING_MANAGER` class.
class Throttler:
"""The default :setting:`THROTTLER` class.
It assigns to each request its domain or subdomain as scope and handles
backoff according to :ref:`backoff settings <basic-throttling>`.
@ -520,13 +520,13 @@ class ThrottlingManager:
self.crawler = crawler
_warn_on_deprecated_concurrency(crawler.settings)
_warn_on_unachievable_concurrency(crawler.settings)
self._debug = crawler.settings.getbool("THROTTLING_DEBUG")
self._max_idle = crawler.settings.getfloat("THROTTLING_SCOPE_MAX_IDLE")
self._debug = crawler.settings.getbool("THROTTLER_DEBUG")
self._max_idle = crawler.settings.getfloat("THROTTLER_SCOPE_MAX_IDLE")
self._robotstxt_obey = crawler.settings.getbool(
"ROBOTSTXT_OBEY"
) and crawler.settings.getbool("THROTTLING_ROBOTSTXT_OBEY")
) and crawler.settings.getbool("THROTTLER_ROBOTSTXT_OBEY")
self._robotstxt_max_delay = crawler.settings.getfloat(
"THROTTLING_ROBOTSTXT_MAX_DELAY"
"THROTTLER_ROBOTSTXT_MAX_DELAY"
)
self._default_useragent: str = crawler.settings["USER_AGENT"]
self._robotstxt_useragent: str | None = crawler.settings["ROBOTSTXT_USER_AGENT"]
@ -535,40 +535,40 @@ class ThrottlingManager:
self._on_robots_parsed, signal=signals.robots_parsed
)
self._default_scope_manager_cls = load_object(
crawler.settings["THROTTLING_SCOPE_MANAGER"]
crawler.settings["THROTTLER_SCOPE_MANAGER"]
)
self._scopes_config: dict[str, dict[str, Any]] = self._merge_download_slots(
crawler.settings
)
# Ordered by least-recently-used first (see get_scope_manager), so the
# scope limit can evict the coldest idle scopes (see THROTTLING_SCOPE_LIMIT).
self._scope_managers: OrderedDict[ScopeID, ThrottlingScopeManagerProtocol] = (
# scope limit can evict the coldest idle scopes (see THROTTLER_SCOPE_LIMIT).
self._scope_managers: OrderedDict[ScopeID, ThrottlerScopeManagerProtocol] = (
OrderedDict()
)
self._scope_limit: int = crawler.settings.getint("THROTTLING_SCOPE_LIMIT")
self._scope_limit: int = crawler.settings.getint("THROTTLER_SCOPE_LIMIT")
self._last_eviction: float | None = None
# Concurrency slots reserved by acquire(), to be released once the
# request finishes downloading.
self._reserved: WeakKeyDictionary[
Request, list[tuple[ThrottlingScopeManagerProtocol, float | None]]
Request, list[tuple[ThrottlerScopeManagerProtocol, float | None]]
] = WeakKeyDictionary()
@staticmethod
def _merge_download_slots(settings: BaseSettings) -> dict[str, dict[str, Any]]:
"""Return the effective per-scope configuration, merging the deprecated
:setting:`DOWNLOAD_SLOTS` setting into :setting:`THROTTLING_SCOPES`.
:setting:`DOWNLOAD_SLOTS` setting into :setting:`THROTTLER_SCOPES`.
Each ``DOWNLOAD_SLOTS`` entry is translated to a throttling scope keyed
Each ``DOWNLOAD_SLOTS`` entry is translated to a throttler scope keyed
by the same slot name (the default manager keys domain scopes by host
name, which is what download slots used too): ``concurrency`` and
``delay`` map directly, and the ``randomize_delay`` boolean maps to a
``jitter`` magnitude (the historical ±50%, or none). An explicit
``THROTTLING_SCOPES`` entry for the same scope takes precedence over the
``THROTTLER_SCOPES`` entry for the same scope takes precedence over the
translated one. The deprecation warning is emitted by the downloader.
"""
scopes: dict[str, dict[str, Any]] = {
scope_id: dict(config)
for scope_id, config in settings.getdict("THROTTLING_SCOPES").items()
for scope_id, config in settings.getdict("THROTTLER_SCOPES").items()
}
for slot_id, slot_config in settings.getdict("DOWNLOAD_SLOTS").items():
translated: dict[str, Any] = {}
@ -578,7 +578,7 @@ class ThrottlingManager:
translated["delay"] = slot_config["delay"]
if "randomize_delay" in slot_config:
translated["jitter"] = 0.5 if slot_config["randomize_delay"] else 0.0
# An explicit THROTTLING_SCOPES entry wins over the translated one.
# An explicit THROTTLER_SCOPES entry wins over the translated one.
scopes[slot_id] = {**translated, **scopes.get(slot_id, {})}
return scopes
@ -597,14 +597,14 @@ class ThrottlingManager:
:func:`scope_cache`). Subclasses whose :meth:`get_scopes` cannot be
resolved synchronously rely on that persisted value instead.
"""
scopes = request.meta.get("throttling_scopes")
scopes = request.meta.get("throttler_scopes")
if scopes is not None:
return cast("RequestScopes", scopes)
download_slot = request.meta.get("download_slot")
if download_slot is not None:
warnings.warn(
"The 'download_slot' request meta key is deprecated. Use "
"'throttling_scopes' instead.",
"'throttler_scopes' instead.",
category=ScrapyDeprecationWarning,
stacklevel=2,
)
@ -641,7 +641,7 @@ class ThrottlingManager:
# -- Scope-state coordination (called from the request lifecycle) --------
def get_scope_manager(self, scope_id: ScopeID) -> ThrottlingScopeManagerProtocol:
def get_scope_manager(self, scope_id: ScopeID) -> ThrottlerScopeManagerProtocol:
manager = self._scope_managers.get(scope_id)
if manager is not None:
# Mark as most-recently-used for the LRU scope limit.
@ -655,7 +655,7 @@ class ThrottlingManager:
else self._default_scope_manager_cls
)
manager = cast(
"ThrottlingScopeManagerProtocol",
"ThrottlerScopeManagerProtocol",
build_from_crawler(manager_cls, self.crawler, config),
)
self._scope_managers[scope_id] = manager
@ -664,7 +664,7 @@ class ThrottlingManager:
def _enforce_scope_limit(self, keep: ScopeID) -> None:
"""Evict least-recently-used idle scopes while the number of live scope
managers exceeds :setting:`THROTTLING_SCOPE_LIMIT` (``0`` disables the
managers exceeds :setting:`THROTTLER_SCOPE_LIMIT` (``0`` disables the
limit).
LRU order is kept by :meth:`get_scope_manager` moving each accessed
@ -684,7 +684,7 @@ class ThrottlingManager:
del self._scope_managers[scope_id]
async def acquire(self, request: Request) -> None:
# A throttling-aware scheduler reserves the request before handing it
# A throttler-aware scheduler reserves the request before handing it
# to the engine, so there is nothing left to wait for or record here.
if request in self._reserved:
return
@ -728,7 +728,7 @@ class ThrottlingManager:
def _record_reservation(
self,
request: Request,
managers: list[tuple[ThrottlingScopeManagerProtocol, float | None]],
managers: list[tuple[ThrottlerScopeManagerProtocol, float | None]],
) -> None:
"""Record a send on each of *request*'s scope *managers* and mark
*request* as reserved, so :meth:`release` can later free the slots. This
@ -744,7 +744,7 @@ class ThrottlingManager:
for manager, _ in managers:
manager.record_done()
# -- Synchronous readiness API (used by a throttling-aware scheduler) ------
# -- Synchronous readiness API (used by a throttler-aware scheduler) ------
def is_ready(self, request: Request) -> bool:
now = time.monotonic()
@ -759,7 +759,7 @@ class ThrottlingManager:
return True
def reserve(self, request: Request) -> None:
# A throttling-aware scheduler reserves every request before handing it
# A throttler-aware scheduler reserves every request before handing it
# to the engine, so acquire() always returns early for it and never
# gets to evict idle scopes; do it here so their managers do not pile
# up on broad crawls.
@ -789,8 +789,8 @@ class ThrottlingManager:
"""Block until any of *managers* frees a concurrency slot.
Each manager hands out an event Deferred that fires when a slot is freed
(via :meth:`ThrottlingScopeManager.record_done`) or the limit is raised
(via :meth:`ThrottlingScopeManager.set_concurrency`). A long safety timer
(via :meth:`ThrottlerScopeManager.record_done`) or the limit is raised
(via :meth:`ThrottlerScopeManager.set_concurrency`). A long safety timer
bounds the wait in case no slot is ever freed (it always should be, via
:meth:`release`).
"""
@ -802,7 +802,7 @@ class ThrottlingManager:
manager.discard_slot_event(event)
async def _delay_request(self, request: Request) -> None:
"""Honor the :reqmeta:`throttling_delay` meta key by holding *request*
"""Honor the :reqmeta:`delay` meta key by holding *request*
for the requested number of seconds the first time it is processed.
This is the blocking (:meth:`acquire`) counterpart of
@ -815,31 +815,31 @@ class ThrottlingManager:
if wait <= 0:
return
await sleep(wait)
request.meta["_throttling_delayed"] = True
request.meta["_throttler_delayed"] = True
def _request_delay_deadline(self, request: Request, now: float) -> float:
"""Return the monotonic time before which *request* must not be sent due
to its :reqmeta:`throttling_delay`, or ``0.0`` if it has none.
to its :reqmeta:`delay`, or ``0.0`` if it has none.
This is the readiness-API counterpart of :meth:`_delay_request`:
a throttling-aware scheduler gates requests through :meth:`is_ready` and
a throttler-aware scheduler gates requests through :meth:`is_ready` and
:meth:`get_time_until_ready` instead of awaiting :meth:`acquire`, so the
delay is enforced by holding back the request until this deadline rather
than by sleeping. The deadline is computed once, the first time the
request reaches the gate, and stored so later polls reuse it.
A request whose delay has already been honored (the ``_throttling_delayed``
A request whose delay has already been honored (the ``_throttler_delayed``
flag, also set by :meth:`_delay_request`) is never delayed again,
which keeps a resumed crawl from re-blocking on a stale deadline."""
delay = request.meta.get("throttling_delay")
if not delay or request.meta.get("_throttling_delayed"):
delay = request.meta.get("delay")
if not delay or request.meta.get("_throttler_delayed"):
return 0.0
deadline = request.meta.get("_throttling_delay_deadline")
deadline = request.meta.get("_throttler_delay_deadline")
if deadline is None:
deadline = now + float(delay)
request.meta["_throttling_delay_deadline"] = deadline
request.meta["_throttler_delay_deadline"] = deadline
if self._debug:
logger.debug(f"Holding {request} for {delay:.2f}s (throttling_delay)")
logger.debug(f"Holding {request} for {delay:.2f}s (delay)")
return deadline
def back_off(
@ -889,11 +889,11 @@ class ThrottlingManager:
def apply_robots_crawl_delay(self, scope_id: ScopeID, delay: float) -> None:
"""Honor a robots.txt ``Crawl-delay`` directive of *delay* seconds for
*scope_id* by setting its delay (capped at
:setting:`THROTTLING_ROBOTSTXT_MAX_DELAY`) and its concurrency to ``1``.
:setting:`THROTTLER_ROBOTSTXT_MAX_DELAY`) and its concurrency to ``1``.
Called from the :signal:`robots_parsed` signal handler when
:setting:`THROTTLING_ROBOTSTXT_OBEY` is enabled. An explicit
:setting:`THROTTLING_SCOPES` configuration for the scope is respected, but
:setting:`THROTTLER_ROBOTSTXT_OBEY` is enabled. An explicit
:setting:`THROTTLER_SCOPES` configuration for the scope is respected, but
a warning is logged about the discrepancy unless its ``ignore_robots_txt``
key is ``True``.
"""
@ -910,10 +910,10 @@ class ThrottlingManager:
conflicts.append(f"concurrency={config['concurrency']!r} > 1")
if conflicts:
logger.warning(
f"Throttling scope {scope_id!r} is configured with {' and '.join(conflicts)}, "
f"Throttler scope {scope_id!r} is configured with {' and '.join(conflicts)}, "
f"which is more aggressive than its robots.txt Crawl-delay of "
f"{capped}s. The configured values take precedence; set "
"'ignore_robots_txt': True in its THROTTLING_SCOPES entry to "
"'ignore_robots_txt': True in its THROTTLER_SCOPES entry to "
"silence this warning."
)
return
@ -951,12 +951,12 @@ class ThrottlingManager:
del self._scope_managers[scope_id]
class ThrottlingScopeManagerProtocol(Protocol):
"""A protocol for :setting:`THROTTLING_SCOPE_MANAGER` :ref:`components
class ThrottlerScopeManagerProtocol(Protocol):
"""A protocol for :setting:`THROTTLER_SCOPE_MANAGER` :ref:`components
<topics-components>`.
The ``__init__`` method gets a ``config`` dict with the base configuration
of the managed throttling scope. For example:
of the managed throttler scope. For example:
.. code-block:: python
@ -988,7 +988,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
"""Return the number of seconds to wait before a request for this scope
may be sent, or ``0`` if it may be sent right away.
*amount* is the expected :ref:`throttling quota <throttling-quotas>`
*amount* is the expected :ref:`throttler quota <throttling-quotas>`
consumption of the request, if any.
"""
@ -996,7 +996,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
self, now: float | None = None, amount: float | None = None
) -> None:
"""Record that a request for this scope has just been sent, consuming
*amount* of its :ref:`throttling quota <throttling-quotas>` if given."""
*amount* of its :ref:`throttler quota <throttling-quotas>` if given."""
def record_done(self, now: float | None = None) -> None:
"""Record that a previously :meth:`record_sent` request has finished
@ -1017,7 +1017,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
*cap* limits *delay* to :setting:`BACKOFF_MAX_DELAY`. It is ``True`` for
untrusted input such as response headers, and may be set to ``False``
for trusted, programmatic delays (see
:meth:`ThrottlingManagerProtocol.delay_scope`).
:meth:`ThrottlerProtocol.delay_scope`).
"""
def reconcile_quota(
@ -1026,7 +1026,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
remaining: float | None = None,
now: float | None = None,
) -> None:
"""Reconcile the :ref:`throttling quota <throttling-quotas>` of this
"""Reconcile the :ref:`throttler quota <throttling-quotas>` of this
scope with the actual *consumed* amount (or the *remaining* amount)
reported for a request, correcting the estimate used by
:meth:`record_sent`."""
@ -1049,7 +1049,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
def concurrency_blocked(self) -> bool:
"""Return whether this scope is at its concurrency limit.
:class:`ThrottlingManager` calls this (after all time-based gates in
:class:`Throttler` calls this (after all time-based gates in
:meth:`can_send` are open) to decide whether to wait for a freed slot.
Return ``False`` when no concurrency limit is enforced.
"""
@ -1058,7 +1058,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
"""Return the current load of this scope: a non-negative number, with
``1.0`` meaning "as busy as its concurrency limit allows".
A :ref:`throttling-aware scheduler <throttling-aware-scheduler>` uses
A :ref:`throttler-aware scheduler <throttler-aware-scheduler>` uses
this to break ties between equally-prioritized requests, preferring the
least-loaded scopes. The reference implementation returns active sends
divided by the concurrency limit (falling back to
@ -1076,7 +1076,7 @@ class ThrottlingScopeManagerProtocol(Protocol):
def discard_slot_event(self, event: Deferred[None]) -> None:
"""Cancel a pending slot event returned by :meth:`slot_event`.
Called by :class:`ThrottlingManager` when the wait ends without the
Called by :class:`Throttler` when the wait ends without the
event firing (e.g. another scope's slot opened first).
"""
@ -1095,8 +1095,8 @@ class ThrottlingScopeManagerProtocol(Protocol):
_SLOT_WAIT_TIMEOUT = 1.0
class ThrottlingScopeManager:
"""The default :setting:`THROTTLING_SCOPE_MANAGER` class.
class ThrottlerScopeManager:
"""The default :setting:`THROTTLER_SCOPE_MANAGER` class.
It implements a per-scope state machine covering delay, exponential
:ref:`backoff <backoff>`, concurrency and :ref:`quotas
@ -1122,7 +1122,7 @@ class ThrottlingScopeManager:
than that many requests are allowed in flight at once.
- When the scope is configured with a ``"quota"``, no more than that much
quota is consumed per ``"window"`` (default: :setting:`THROTTLING_WINDOW`).
quota is consumed per ``"window"`` (default: :setting:`THROTTLER_WINDOW`).
"""
@classmethod
@ -1134,7 +1134,7 @@ class ThrottlingScopeManager:
backoff: dict[str, Any] = config.get("backoff", {})
self._id: ScopeID = config.get("id", "")
# The per-scope delay defaults to DOWNLOAD_DELAY; a scope can override
# it with its own "delay" config (see THROTTLING_SCOPES).
# it with its own "delay" config (see THROTTLER_SCOPES).
self._base_delay: float = float(
config.get("delay", settings.getfloat("DOWNLOAD_DELAY"))
)
@ -1180,7 +1180,7 @@ class ThrottlingScopeManager:
quota = config.get("quota")
self._quota: float | None = None if quota is None else float(quota)
self._quota_window: float = float(
config.get("window", settings.getfloat("THROTTLING_WINDOW"))
config.get("window", settings.getfloat("THROTTLER_WINDOW"))
)
# State.

4
scrapy/utils/test.py
View File

@ -75,9 +75,9 @@ def get_crawler(
if prevent_warnings:
# Pin the per-scope concurrency to CONCURRENT_REQUESTS_PER_DOMAIN's
# default: keeps the suite's historical behavior and silences the
# warn-then-flip warning (see throttling._warn_on_deprecated_concurrency).
# warn-then-flip warning (see throttler._warn_on_deprecated_concurrency).
settings.setdefault(
"THROTTLING_SCOPE_CONCURRENCY",
"THROTTLER_SCOPE_CONCURRENCY",
default_settings.CONCURRENT_REQUESTS_PER_DOMAIN,
)
runner: CrawlerRunnerBase

14
tests/test_crawler.py
View File

@ -55,12 +55,12 @@ def get_raw_crawler(
class TestScopeConcurrencyBridge:
"""CONCURRENT_REQUESTS_PER_DOMAIN is deprecated and bridged into
THROTTLING_SCOPE_CONCURRENCY by setting priority (see
scrapy.throttling._default_scope_concurrency)."""
THROTTLER_SCOPE_CONCURRENCY by setting priority (see
scrapy.throttler._default_scope_concurrency)."""
@staticmethod
def _resolve(settings: Settings) -> int:
from scrapy.throttling import _default_scope_concurrency # noqa: PLC0415
from scrapy.throttler import _default_scope_concurrency # noqa: PLC0415
return _default_scope_concurrency(settings)
@ -75,18 +75,18 @@ class TestScopeConcurrencyBridge:
def test_scope_concurrency_overrides(self) -> None:
settings = Settings()
settings.set("THROTTLING_SCOPE_CONCURRENCY", 4, priority="project")
settings.set("THROTTLER_SCOPE_CONCURRENCY", 4, priority="project")
assert self._resolve(settings) == 4
def test_new_setting_wins_on_non_default_tie(self) -> None:
settings = Settings()
settings.set("CONCURRENT_REQUESTS_PER_DOMAIN", 4, priority="project")
settings.set("THROTTLING_SCOPE_CONCURRENCY", 9, priority="project")
settings.set("THROTTLER_SCOPE_CONCURRENCY", 9, priority="project")
assert self._resolve(settings) == 9
def test_higher_priority_deprecated_setting_wins(self) -> None:
settings = Settings()
settings.set("THROTTLING_SCOPE_CONCURRENCY", 9, priority="project")
settings.set("THROTTLER_SCOPE_CONCURRENCY", 9, priority="project")
settings.set("CONCURRENT_REQUESTS_PER_DOMAIN", 4, priority="cmdline")
assert self._resolve(settings) == 4
@ -111,7 +111,7 @@ class TestScopeConcurrencyBridge:
with warnings.catch_warnings(record=True) as recorded:
warnings.simplefilter("always")
get_crawler(
settings_dict={"THROTTLING_SCOPE_CONCURRENCY": 4},
settings_dict={"THROTTLER_SCOPE_CONCURRENCY": 4},
prevent_warnings=False,
)
messages = [str(w.message) for w in recorded]

26
tests/test_engine.py
View File

@ -655,7 +655,7 @@ async def test_request_scheduled_signal():
crawler.signals.disconnect(signal_handler, signals.request_scheduled)
class TestEngineThrottling:
class TestEngineThrottler:
@pytest.fixture
def engine(self):
crawler = get_crawler(MySpider)
@ -708,25 +708,25 @@ class TestEngineThrottling:
engine._maybe_arm_delay_wakeup()
assert engine._delay_wakeup is None
def test_maybe_warn_throttling_backout(self, engine):
# A scheduler without get_next_request_delay is not throttling-aware, so
def test_maybe_warn_throttler_backout(self, engine):
# A scheduler without get_next_request_delay is not throttler-aware, so
# the warning recommends switching to one.
engine._get_next_request_delay = None
with LogCapture() as log:
engine._maybe_warn_throttling_backout()
engine._maybe_warn_throttler_backout()
# A second call is a no-op (the warning is emitted only once).
engine._maybe_warn_throttling_backout()
assert engine._throttling_backout_warned is True
assert str(log).count("ThrottlingAwareScheduler") == 1
engine._maybe_warn_throttler_backout()
assert engine._throttler_backout_warned is True
assert str(log).count("ThrottlerAwareScheduler") == 1
def test_maybe_warn_throttling_backout_throttling_aware(self, engine):
# A throttling-aware scheduler (one with get_next_request_delay) holds
def test_maybe_warn_throttler_backout_throttler_aware(self, engine):
# A throttler-aware scheduler (one with get_next_request_delay) holds
# throttled requests itself, so no warning is emitted.
engine._get_next_request_delay = lambda: None
with LogCapture() as log:
engine._maybe_warn_throttling_backout()
assert engine._throttling_backout_warned is False
assert "ThrottlingAwareScheduler" not in str(log)
engine._maybe_warn_throttler_backout()
assert engine._throttler_backout_warned is False
assert "ThrottlerAwareScheduler" not in str(log)
def test_spider_is_idle_false_while_scheduling(self, engine):
engine._slot = Mock()
@ -734,7 +734,7 @@ class TestEngineThrottling:
engine.scraper.slot.is_idle.return_value = True
engine.downloader = Mock()
engine.downloader.active = []
engine._throttling_waiting = set()
engine._throttler_waiting = set()
engine._start = None
engine._scheduling = 1
# An in-flight async enqueue keeps the spider from being considered idle.

103
tests/test_pqueues.py
View File

@ -8,11 +8,11 @@ from scrapy.http.request import Request
from scrapy.pqueues import (
DownloaderAwarePriorityQueue,
ScrapyPriorityQueue,
ThrottlingAwarePriorityQueue,
ThrottlerAwarePriorityQueue,
)
from scrapy.spiders import Spider
from scrapy.squeues import FifoMemoryQueue, PickleFifoDiskQueue
from scrapy.throttling import iter_scopes
from scrapy.throttler import iter_scopes
from scrapy.utils.misc import build_from_crawler, load_object
from scrapy.utils.test import get_crawler
from tests.test_scheduler import MockDownloader
@ -191,22 +191,22 @@ class TestDownloaderAwarePriorityQueue:
# No active downloads are tracked in the downloader, so every slot has
# the same score and tie-breaking must not starve a slot.
req_a1 = Request("https://example.org/a1")
req_a1.meta["throttling_scopes"] = "slot-a"
req_a1.meta["throttler_scopes"] = "slot-a"
req_b1 = Request("https://example.org/b1")
req_b1.meta["throttling_scopes"] = "slot-b"
req_b1.meta["throttler_scopes"] = "slot-b"
req_a2 = Request("https://example.org/a2")
req_a2.meta["throttling_scopes"] = "slot-a"
req_a2.meta["throttler_scopes"] = "slot-a"
req_b2 = Request("https://example.org/b2")
req_b2.meta["throttling_scopes"] = "slot-b"
req_b2.meta["throttler_scopes"] = "slot-b"
for request in (req_a1, req_b1, req_a2, req_b2):
self.queue.push(request)
slots = [
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
]
assert slots == ["slot-a", "slot-b", "slot-a", "slot-b"]
@ -215,22 +215,22 @@ class TestDownloaderAwarePriorityQueue:
# If the selected slot becomes empty, rotation should continue from
# that slot marker to avoid restarting from the smallest slot.
req_a1 = Request("https://example.org/a1")
req_a1.meta["throttling_scopes"] = "slot-a"
req_a1.meta["throttler_scopes"] = "slot-a"
req_a2 = Request("https://example.org/a2")
req_a2.meta["throttling_scopes"] = "slot-a"
req_a2.meta["throttler_scopes"] = "slot-a"
req_b1 = Request("https://example.org/b1")
req_b1.meta["throttling_scopes"] = "slot-b"
req_b1.meta["throttler_scopes"] = "slot-b"
req_c1 = Request("https://example.org/c1")
req_c1.meta["throttling_scopes"] = "slot-c"
req_c1.meta["throttler_scopes"] = "slot-c"
for request in (req_a1, req_a2, req_b1, req_c1):
self.queue.push(request)
slots = [
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttling_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
self.queue.pop().meta["throttler_scopes"],
]
assert slots == ["slot-a", "slot-b", "slot-c", "slot-a"]
@ -240,11 +240,11 @@ class TestDownloaderAwarePriorityQueue:
assert throttler is not None
req_a = Request("https://example.org/a")
req_a.meta["throttling_scopes"] = "slot-a"
req_a.meta["throttler_scopes"] = "slot-a"
req_b = Request("https://example.org/b")
req_b.meta["throttling_scopes"] = "slot-b"
req_b.meta["throttler_scopes"] = "slot-b"
req_c = Request("https://example.org/c")
req_c.meta["throttling_scopes"] = "slot-c"
req_c.meta["throttler_scopes"] = "slot-c"
for req in (req_a, req_b, req_c):
self.queue.push(req)
@ -257,7 +257,7 @@ class TestDownloaderAwarePriorityQueue:
def test_contains(self):
req = Request("https://example.org/")
req.meta["throttling_scopes"] = "example-slot"
req.meta["throttler_scopes"] = "example-slot"
assert "example-slot" not in self.queue
self.queue.push(req)
assert "example-slot" in self.queue
@ -317,10 +317,10 @@ def test_pop_order(input_, output):
assert actual_output_urls == expected_output_urls
class TestThrottlingAwarePriorityQueue:
class TestThrottlerAwarePriorityQueue:
def _queue(self, crawler, key=""):
return build_from_crawler(
ThrottlingAwarePriorityQueue,
ThrottlerAwarePriorityQueue,
crawler,
downstream_queue_cls=FifoMemoryQueue,
key=key,
@ -346,7 +346,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(
Spider,
settings_dict={
"THROTTLING_SCOPES": {"slow.com": {"delay": 1000.0}},
"THROTTLER_SCOPES": {"slow.com": {"delay": 1000.0}},
"RANDOMIZE_DOWNLOAD_DELAY": False,
},
)
@ -369,13 +369,13 @@ class TestThrottlingAwarePriorityQueue:
assert delay == pytest.approx(1000.0, abs=1.0)
@coroutine_test
async def test_pop_holds_request_with_throttling_delay(self):
async def test_pop_holds_request_with_delay(self):
crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False})
queue = self._queue(crawler)
await self._push(
queue,
crawler,
Request("http://slow.com/1", meta={"throttling_delay": 1000.0}),
Request("http://slow.com/1", meta={"delay": 1000.0}),
)
await self._push(queue, crawler, Request("http://fast.com/1"))
# The delayed request is held back even though its scope is otherwise
@ -398,7 +398,7 @@ class TestThrottlingAwarePriorityQueue:
await self._push(
queue,
crawler,
Request("http://example.com/slow", meta={"throttling_delay": 1000.0}),
Request("http://example.com/slow", meta={"delay": 1000.0}),
)
await self._push(queue, crawler, Request("http://example.com/fast"))
# The delayed request is held aside, so the other request in the same
@ -413,7 +413,7 @@ class TestThrottlingAwarePriorityQueue:
async def test_delayed_request_promoted_when_due(self):
crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False})
queue = self._queue(crawler)
request = Request("http://example.com/slow", meta={"throttling_delay": 1000.0})
request = Request("http://example.com/slow", meta={"delay": 1000.0})
await self._push(queue, crawler, request)
assert queue.pop() is None # held back by its per-request delay
# Once the delay elapses the request is promoted into its scope-set
@ -421,7 +421,7 @@ class TestThrottlingAwarePriorityQueue:
queue._promote_ready(queue._delayed[0][0])
popped = queue.pop()
assert popped is request
assert popped.meta["_throttling_delayed"] is True
assert popped.meta["_throttler_delayed"] is True
assert len(queue) == 0
@coroutine_test
@ -432,7 +432,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(Spider, settings_dict={"RANDOMIZE_DOWNLOAD_DELAY": False})
temp_dir = tempfile.mkdtemp()
queue = build_from_crawler(
ThrottlingAwarePriorityQueue,
ThrottlerAwarePriorityQueue,
crawler,
downstream_queue_cls=PickleFifoDiskQueue,
key=temp_dir,
@ -440,13 +440,13 @@ class TestThrottlingAwarePriorityQueue:
await self._push(
queue,
crawler,
Request("http://example.com/slow", meta={"throttling_delay": 1000.0}),
Request("http://example.com/slow", meta={"delay": 1000.0}),
)
assert len(queue) == 1 # held in memory, not yet in any scope-set queue
state = queue.close() # graceful stop
resumed = build_from_crawler(
ThrottlingAwarePriorityQueue,
ThrottlerAwarePriorityQueue,
crawler,
downstream_queue_cls=PickleFifoDiskQueue,
key=temp_dir,
@ -457,7 +457,7 @@ class TestThrottlingAwarePriorityQueue:
assert popped is not None
assert popped.url == "http://example.com/slow"
# Its delay is marked consumed, so it does not re-block on resume.
assert popped.meta["_throttling_delayed"] is True
assert popped.meta["_throttler_delayed"] is True
resumed.close()
@coroutine_test
@ -465,7 +465,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(
Spider,
settings_dict={
"THROTTLING_SCOPES": {
"THROTTLER_SCOPES": {
"a.com": {"concurrency": 4},
"b.com": {"concurrency": 4},
}
@ -486,7 +486,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(
Spider,
settings_dict={
"THROTTLING_SCOPES": {
"THROTTLER_SCOPES": {
"a.com": {"concurrency": 4},
"b.com": {"concurrency": 4},
}
@ -512,18 +512,6 @@ class TestThrottlingAwarePriorityQueue:
await self._push(queue, crawler, Request("http://a.com/1"))
assert queue.close() != {}
@coroutine_test
async def test_peek(self):
crawler = get_crawler(Spider)
queue = self._queue(crawler)
assert queue.peek() is None
await self._push(queue, crawler, Request("http://a.com/1"))
head = queue.peek()
assert head is not None
assert head.url == "http://a.com/1"
# peek() does not consume the request.
assert len(queue) == 1
@coroutine_test
async def test_get_next_request_delay_zero_when_ready(self):
crawler = get_crawler(Spider)
@ -537,7 +525,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(Spider)
queue = self._queue(crawler)
# An empty (but still registered) internal queue is skipped.
queue.pqueues[frozenset({"a.com"})] = queue.pqfactory(frozenset({"a.com"}))
queue.pqueues[frozenset({"a.com"})] = queue._pqfactory(frozenset({"a.com"}))
assert queue.get_next_request_delay() is None
@coroutine_test
@ -545,7 +533,7 @@ class TestThrottlingAwarePriorityQueue:
crawler = get_crawler(
Spider,
settings_dict={
"THROTTLING_SCOPES": {
"THROTTLER_SCOPES": {
"a.com": {"delay": 10.0},
"b.com": {"delay": 1000.0},
},
@ -572,27 +560,16 @@ class TestThrottlingAwarePriorityQueue:
queue = self._queue(crawler)
await self._push(queue, crawler, Request("http://a.com/1"))
inner = next(iter(queue.pqueues.values()))
# peek() still reports a sendable head, but pop() yields nothing: the
# _select() still reports a sendable head, but pop() yields nothing: the
# request-is-None guard must not try to reserve a missing request.
inner.pop = lambda: None
assert queue.pop() is None
@coroutine_test
async def test_contains(self):
crawler = get_crawler(Spider)
queue = self._queue(crawler)
scope_set = frozenset(
iter_scopes(await crawler.throttler.get_scopes(Request("http://a.com/1")))
)
assert scope_set not in queue
await self._push(queue, crawler, Request("http://a.com/1"))
assert scope_set in queue
def test_non_dict_slot_startprios(self):
crawler = get_crawler(Spider)
with pytest.raises(ValueError, match="slot_startprios"):
build_from_crawler(
ThrottlingAwarePriorityQueue,
ThrottlerAwarePriorityQueue,
crawler,
downstream_queue_cls=FifoMemoryQueue,
key="",

44
tests/test_scheduler.py
View File

@ -11,7 +11,7 @@ from unittest.mock import Mock
import pytest
from scrapy.core.downloader import Downloader
from scrapy.core.scheduler import BaseScheduler, Scheduler, ThrottlingAwareScheduler
from scrapy.core.scheduler import BaseScheduler, Scheduler, ThrottlerAwareScheduler
from scrapy.crawler import Crawler
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.http import Request
@ -411,22 +411,22 @@ class TestIncompatibility:
self._incompatible()
_THROTTLING_AWARE_PQ = "scrapy.pqueues.ThrottlingAwarePriorityQueue"
_THROTTLER_AWARE_PQ = "scrapy.pqueues.ThrottlerAwarePriorityQueue"
class TestThrottlingAwareScheduler:
class TestThrottlerAwareScheduler:
def _crawler(self, settings_dict: dict[str, Any] | None = None) -> Crawler:
settings = {
"SCHEDULER_PRIORITY_QUEUE": _THROTTLING_AWARE_PQ,
"SCHEDULER_PRIORITY_QUEUE": _THROTTLER_AWARE_PQ,
"DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
**(settings_dict or {}),
}
return get_crawler(Spider, settings)
def _scheduler(self, crawler: Crawler) -> ThrottlingAwareScheduler:
def _scheduler(self, crawler: Crawler) -> ThrottlerAwareScheduler:
spider = Spider(name="spider")
crawler.spider = spider
scheduler = ThrottlingAwareScheduler.from_crawler(crawler)
scheduler = ThrottlerAwareScheduler.from_crawler(crawler)
scheduler.open(spider)
return scheduler
@ -449,21 +449,21 @@ class TestThrottlingAwareScheduler:
scheduler.enqueue_request(Request("http://a.com/1"))
scheduler.close("finished")
def test_requires_throttling_aware_priority_queue(self) -> None:
def test_requires_throttler_aware_priority_queue(self) -> None:
crawler = self._crawler(
{"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ScrapyPriorityQueue"}
)
spider = Spider(name="spider")
crawler.spider = spider
scheduler = ThrottlingAwareScheduler.from_crawler(crawler)
with pytest.raises(ValueError, match="throttling-aware priority queue"):
scheduler = ThrottlerAwareScheduler.from_crawler(crawler)
with pytest.raises(ValueError, match="throttler-aware priority queue"):
scheduler.open(spider)
@coroutine_test
async def test_delay_blocks_and_reports_delay(self) -> None:
crawler = self._crawler(
{
"THROTTLING_SCOPES": {"slow.com": {"delay": 1000.0}},
"THROTTLER_SCOPES": {"slow.com": {"delay": 1000.0}},
"RANDOMIZE_DOWNLOAD_DELAY": False,
},
)
@ -481,7 +481,7 @@ class TestThrottlingAwareScheduler:
@coroutine_test
async def test_no_delay_when_only_concurrency_blocked(self) -> None:
crawler = self._crawler(
{"THROTTLING_SCOPES": {"slow.com": {"concurrency": 1}}},
{"THROTTLER_SCOPES": {"slow.com": {"concurrency": 1}}},
)
scheduler = self._scheduler(crawler)
await scheduler.enqueue_request_async(Request("http://slow.com/1"))
@ -494,14 +494,14 @@ class TestThrottlingAwareScheduler:
@coroutine_test
async def test_delayed_request_survives_jobdir_stop(self, tmp_path: Path) -> None:
# A request held back by its per-request throttling_delay must not be
# A request held back by its per-request delay must not be
# lost on a graceful stop when a JOBDIR is configured: it is flushed to
# the disk queue on close and restored on resume.
crawler = self._crawler(
{"JOBDIR": str(tmp_path), "RANDOMIZE_DOWNLOAD_DELAY": False}
)
scheduler = self._scheduler(crawler)
request = Request("http://a.com/slow", meta={"throttling_delay": 1000.0})
request = Request("http://a.com/slow", meta={"delay": 1000.0})
assert await scheduler.enqueue_request_async(request) is True
assert len(scheduler) == 1
# The delay holds it back, so nothing is dequeued before the stop.
@ -585,14 +585,14 @@ class TestThrottlingAwareScheduler:
scheduler2.close("finished")
class TestIntegrationWithThrottlingAwareScheduler:
class TestIntegrationWithThrottlerAwareScheduler:
@inline_callbacks_test
def test_integration(self):
crawler = get_crawler(
spidercls=StartUrlsSpider,
settings_dict={
"SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue",
"SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue",
"DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
},
)
@ -628,8 +628,8 @@ class TestIntegrationWithThrottlingAwareScheduler:
crawler = get_crawler(
spidercls=FollowSpider,
settings_dict={
"SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue",
"SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue",
"DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
},
)
@ -638,16 +638,16 @@ class TestIntegrationWithThrottlingAwareScheduler:
@inline_callbacks_test
def test_integration_with_delay(self):
# A small per-scope delay forces the engine to arm the throttling wakeup
# A small per-scope delay forces the engine to arm the throttler wakeup
# timer between requests; the crawl must still complete.
crawler = get_crawler(
spidercls=StartUrlsSpider,
settings_dict={
"SCHEDULER": "scrapy.core.scheduler.ThrottlingAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlingAwarePriorityQueue",
"SCHEDULER": "scrapy.core.scheduler.ThrottlerAwareScheduler",
"SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.ThrottlerAwarePriorityQueue",
"DUPEFILTER_CLASS": "scrapy.dupefilters.BaseDupeFilter",
"RANDOMIZE_DOWNLOAD_DELAY": False,
"THROTTLING_SCOPES": {"127.0.0.1": {"delay": 0.05}},
"THROTTLER_SCOPES": {"127.0.0.1": {"delay": 0.05}},
},
)
with MockServer() as mockserver:

156
tests/test_throttling.py → tests/test_throttler.py
View File

@ -8,9 +8,9 @@ from scrapy import signals
from scrapy.exceptions import DownloadTimeoutError
from scrapy.http import Request, Response
from scrapy.settings import default_settings
from scrapy.throttling import (
ThrottlingManager,
ThrottlingScopeManager,
from scrapy.throttler import (
Throttler,
ThrottlerScopeManager,
_add_bare_scope,
_parse_ratelimit_reset,
_parse_retry_after,
@ -30,12 +30,12 @@ from tests.utils.decorators import coroutine_test
def _manager(settings=None):
crawler = get_crawler(settings_dict=settings)
return ThrottlingManager.from_crawler(crawler)
return Throttler.from_crawler(crawler)
def _scope_manager(settings=None, config=None):
crawler = get_crawler(settings_dict=settings)
return ThrottlingScopeManager.from_crawler(crawler, config or {"id": "example.com"})
return ThrottlerScopeManager.from_crawler(crawler, config or {"id": "example.com"})
class _FakeRobotParser:
@ -67,11 +67,11 @@ def test_deprecated_concurrency_defaults_differ():
shipping a bogus warning or aborting a crawl."""
assert (
default_settings.CONCURRENT_REQUESTS_PER_DOMAIN
!= default_settings.THROTTLING_SCOPE_CONCURRENCY
!= default_settings.THROTTLER_SCOPE_CONCURRENCY
)
class TestThrottlingManager:
class TestThrottler:
@coroutine_test
async def test_get_scopes_returns_netloc(self):
manager = _manager()
@ -91,20 +91,20 @@ class TestThrottlingManager:
@coroutine_test
async def test_get_scopes_meta_string(self):
manager = _manager()
request = Request("http://example.com/a", meta={"throttling_scopes": "api"})
request = Request("http://example.com/a", meta={"throttler_scopes": "api"})
assert await manager.get_scopes(request) == "api"
@coroutine_test
async def test_get_scopes_meta_dict(self):
manager = _manager()
request = Request(
"http://example.com/a", meta={"throttling_scopes": {"api": 2.0}}
"http://example.com/a", meta={"throttler_scopes": {"api": 2.0}}
)
assert await manager.get_scopes(request) == {"api": 2.0}
@coroutine_test
async def test_get_scopes_persisted_in_meta(self):
from scrapy.throttling import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415
from scrapy.throttler import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415
manager = _manager()
request = Request("http://example.com/a")
@ -114,7 +114,7 @@ class TestThrottlingManager:
@coroutine_test
async def test_scope_cache_works_without_crawler(self):
# scope_cache only persists to meta; it needs nothing from the manager.
from scrapy.throttling import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415
from scrapy.throttler import _RESOLVED_SCOPES_META_KEY # noqa: PLC0415
class CrawlerlessManager:
@scope_cache
@ -131,7 +131,7 @@ class TestThrottlingManager:
# path reuses them instead of resolving again.
calls = []
class CountingManager(ThrottlingManager):
class CountingManager(Throttler):
@scope_cache
async def get_scopes(self, request):
calls.append(request.url)
@ -155,7 +155,7 @@ class TestThrottlingManager:
manager = _manager()
request = Request(
"http://example.com/a", meta={"throttling_scopes": {"bucket": 3.0}}
"http://example.com/a", meta={"throttler_scopes": {"bucket": 3.0}}
)
await manager.get_scopes(request)
# A request restored from a disk queue is a fresh object; the synchronous
@ -183,13 +183,13 @@ class TestThrottlingManager:
def test_scope_manager_class_in_config(self):
manager = _manager(
{"THROTTLING_SCOPES": {"example.com": {"manager": ThrottlingScopeManager}}}
{"THROTTLER_SCOPES": {"example.com": {"manager": ThrottlerScopeManager}}}
)
scope = manager._get_scope_manager("example.com")
assert isinstance(scope, ThrottlingScopeManager)
assert isinstance(scope, ThrottlerScopeManager)
def test_release_frees_concurrency(self):
manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}})
manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}})
scope = manager._get_scope_manager("example.com")
request = Request("http://example.com")
scope.record_sent(now=0.0)
@ -205,7 +205,7 @@ class TestThrottlingManager:
async def test_acquire_waits_for_freed_slot(self):
from scrapy.utils.asyncio import call_later # noqa: PLC0415
manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}})
manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}})
r1 = Request("http://example.com/1")
r2 = Request("http://example.com/2")
# Drive acquire() the way the engine does, so it runs as a real task that
@ -222,7 +222,7 @@ class TestThrottlingManager:
assert r2 in manager._reserved
def test_apply_backoff_reconciles_quota_without_backoff(self):
manager = _manager({"THROTTLING_SCOPES": {"cost": {"quota": 100.0}}})
manager = _manager({"THROTTLER_SCOPES": {"cost": {"quota": 100.0}}})
scope = manager._get_scope_manager("cost")
manager._apply_backoff({"cost": {"consumed": 5.0}})
# Quota was reconciled but no backoff step was applied.
@ -230,7 +230,7 @@ class TestThrottlingManager:
assert scope._backoff_level == 0
def test_apply_backoff_delay_and_consumed(self):
manager = _manager({"THROTTLING_SCOPES": {"cost": {"quota": 100.0}}})
manager = _manager({"THROTTLER_SCOPES": {"cost": {"quota": 100.0}}})
scope = manager._get_scope_manager("cost")
manager._apply_backoff({"cost": {"delay": 5.0, "consumed": 2.0}})
assert scope._consumed == pytest.approx(2.0)
@ -317,7 +317,7 @@ class TestThrottlingManager:
("settings", "parser_delay", "expected_base_delay"),
[
({"ROBOTSTXT_OBEY": True, "RANDOMIZE_DOWNLOAD_DELAY": False}, 3.0, 3.0),
({"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_OBEY": False}, 3.0, 0.0),
({"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_OBEY": False}, 3.0, 0.0),
({"ROBOTSTXT_OBEY": True}, None, 0.0),
({"ROBOTSTXT_OBEY": True}, ValueError(), 0.0),
],
@ -344,13 +344,13 @@ class TestThrottlingManager:
def test_apply_robots_crawl_delay_capped(self):
manager = _manager(
{"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_MAX_DELAY": 2.0}
{"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_MAX_DELAY": 2.0}
)
manager.apply_robots_crawl_delay("example.com", 30.0)
assert manager._get_scope_manager("example.com")._base_delay == 2.0
def test_apply_robots_crawl_delay_disabled(self):
manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLING_ROBOTSTXT_OBEY": False})
manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLER_ROBOTSTXT_OBEY": False})
manager.apply_robots_crawl_delay("example.com", 3.0)
assert manager._get_scope_manager("example.com")._base_delay == 0.0
@ -363,10 +363,10 @@ class TestThrottlingManager:
manager = _manager(
{
"ROBOTSTXT_OBEY": True,
"THROTTLING_SCOPES": {"example.com": {"concurrency": 8}},
"THROTTLER_SCOPES": {"example.com": {"concurrency": 8}},
}
)
with caplog.at_level(logging.WARNING, logger="scrapy.throttling"):
with caplog.at_level(logging.WARNING, logger="scrapy.throttler"):
manager.apply_robots_crawl_delay("example.com", 3.0)
assert "Crawl-delay" in caplog.text
# The configured value takes precedence (crawl-delay not applied).
@ -376,19 +376,19 @@ class TestThrottlingManager:
manager = _manager(
{
"ROBOTSTXT_OBEY": True,
"THROTTLING_SCOPES": {
"THROTTLER_SCOPES": {
"example.com": {"concurrency": 8, "ignore_robots_txt": True}
},
}
)
with caplog.at_level(logging.WARNING, logger="scrapy.throttling"):
with caplog.at_level(logging.WARNING, logger="scrapy.throttler"):
manager.apply_robots_crawl_delay("example.com", 3.0)
# No warning is logged and the crawl-delay is not applied.
assert "Crawl-delay" not in caplog.text
assert manager._get_scope_manager("example.com")._base_delay == 0.0
def test_scope_eviction(self):
manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 100.0})
manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 100.0})
scope = manager._get_scope_manager("example.com")
scope.record_sent(now=0.0)
scope.record_done(now=0.0)
@ -403,22 +403,22 @@ class TestThrottlingManager:
def test_scope_eviction_skips_active_backoff(self):
manager = _manager(
{"THROTTLING_SCOPE_MAX_IDLE": 100.0, "BACKOFF_MAX_DELAY": 100_000.0}
{"THROTTLER_SCOPE_MAX_IDLE": 100.0, "BACKOFF_MAX_DELAY": 100_000.0}
)
scope = manager._get_scope_manager("example.com")
scope.record_backoff(delay=10_000.0, now=0.0)
manager._last_eviction = None
manager._maybe_evict(now=5_000.0)
# Still in backoff (in_backoff_until far in the future), so not evicted
# even though it has been idle for longer than THROTTLING_SCOPE_MAX_IDLE.
# even though it has been idle for longer than THROTTLER_SCOPE_MAX_IDLE.
assert "example.com" in manager._scope_managers
def test_reserve_evicts_idle_scopes(self):
# A throttling-aware scheduler reserves every request before the engine
# A throttler-aware scheduler reserves every request before the engine
# reaches acquire() (which fast-paths reserved requests), so reserve()
# must be the hook that evicts idle scope managers; otherwise they pile
# up unbounded on broad crawls.
manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 1.0})
manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 1.0})
idle = manager._get_scope_manager("idle.example")
# Make it look long-idle: a finished send in the distant monotonic past.
idle.record_sent(now=0.0)
@ -429,7 +429,7 @@ class TestThrottlingManager:
assert "active.example" in manager._scope_managers
def test_scope_limit_evicts_least_recently_used(self):
manager = _manager({"THROTTLING_SCOPE_LIMIT": 2})
manager = _manager({"THROTTLER_SCOPE_LIMIT": 2})
# Use three scopes in order; each send/done leaves them idle.
for scope_id in ("a.example", "b.example", "c.example"):
scope = manager._get_scope_manager(scope_id)
@ -439,7 +439,7 @@ class TestThrottlingManager:
assert set(manager._scope_managers) == {"b.example", "c.example"}
def test_scope_limit_keeps_active_scopes(self):
manager = _manager({"THROTTLING_SCOPE_LIMIT": 1})
manager = _manager({"THROTTLER_SCOPE_LIMIT": 1})
# Two scopes with in-flight requests cannot be evicted, so the limit is
# exceeded rather than dropping a scope that still tracks a live send.
for scope_id in ("a.example", "b.example"):
@ -447,7 +447,7 @@ class TestThrottlingManager:
assert set(manager._scope_managers) == {"a.example", "b.example"}
def test_scope_limit_disabled(self):
manager = _manager({"THROTTLING_SCOPE_LIMIT": 0})
manager = _manager({"THROTTLER_SCOPE_LIMIT": 0})
for i in range(5):
scope = manager._get_scope_manager(f"{i}.example")
scope.record_sent(now=0.0)
@ -455,7 +455,7 @@ class TestThrottlingManager:
assert len(manager._scope_managers) == 5
class TestThrottlingScopeManager:
class TestThrottlerScopeManager:
def test_no_delay_by_default(self):
scope = _scope_manager()
scope.record_sent(now=0.0)
@ -588,10 +588,10 @@ class TestThrottlingScopeManager:
assert scope._concurrency == 8
def test_no_scope_concurrency_limit_when_zero(self):
# THROTTLING_SCOPE_CONCURRENCY governs scopes that are neither a domain
# THROTTLER_SCOPE_CONCURRENCY governs scopes that are neither a domain
# nor an IP (here a bare "custom" group name).
scope = _scope_manager(
settings={"THROTTLING_SCOPE_CONCURRENCY": 0}, config={"id": "custom"}
settings={"THROTTLER_SCOPE_CONCURRENCY": 0}, config={"id": "custom"}
)
assert scope._concurrency is None
for _ in range(100):
@ -702,8 +702,8 @@ class TestThrottlingScopeManager:
assert scope._consumed == pytest.approx(7.0)
class TestThrottlingManagerReadiness:
"""The synchronous readiness API used by a throttling-aware scheduler."""
class TestThrottlerReadiness:
"""The synchronous readiness API used by a throttler-aware scheduler."""
@coroutine_test
async def test_is_ready_unconstrained_scope(self):
@ -725,7 +725,7 @@ class TestThrottlingManagerReadiness:
async def test_reserve_blocks_delay_scope(self):
manager = _manager(
{
"THROTTLING_SCOPES": {"example.com": {"delay": 100.0}},
"THROTTLER_SCOPES": {"example.com": {"delay": 100.0}},
"RANDOMIZE_DOWNLOAD_DELAY": False,
}
)
@ -740,28 +740,28 @@ class TestThrottlingManagerReadiness:
assert manager.get_time_until_ready(second) == pytest.approx(100.0, abs=1.0)
@coroutine_test
async def test_throttling_delay_blocks_until_deadline(self):
manager = _manager({"THROTTLING_DEBUG": True})
request = Request("http://example.com/a", meta={"throttling_delay": 100.0})
async def test_delay_blocks_until_deadline(self):
manager = _manager({"THROTTLER_DEBUG": True})
request = Request("http://example.com/a", meta={"delay": 100.0})
await manager.get_scopes(request)
# The per-request delay holds back the request even though its scope is
# otherwise unconstrained.
assert manager.is_ready(request) is False
assert manager.get_time_until_ready(request) == pytest.approx(100.0, abs=1.0)
# The deadline is computed once and reused by later polls.
deadline = request.meta["_throttling_delay_deadline"]
deadline = request.meta["_throttler_delay_deadline"]
assert manager.is_ready(request) is False
assert request.meta["_throttling_delay_deadline"] == deadline
assert request.meta["_throttler_delay_deadline"] == deadline
@coroutine_test
async def test_throttling_delay_not_reapplied_once_consumed(self):
async def test_delay_not_reapplied_once_consumed(self):
# A request whose delay was already honored (e.g. promoted out of a
# throttling-aware queue's holding area, or restored on resume) is
# throttler-aware queue's holding area, or restored on resume) is
# ready, so it cannot re-block its scope set on a stale deadline.
manager = _manager()
request = Request(
"http://example.com/a",
meta={"throttling_delay": 100.0, "_throttling_delayed": True},
meta={"delay": 100.0, "_throttler_delayed": True},
)
await manager.get_scopes(request)
assert manager.is_ready(request) is True
@ -771,16 +771,14 @@ class TestThrottlingManagerReadiness:
async def test_get_request_delay(self):
manager = _manager()
assert manager.get_request_delay(
Request("http://example.com/a", meta={"throttling_delay": 100.0})
Request("http://example.com/a", meta={"delay": 100.0})
) == pytest.approx(100.0, abs=1.0)
# A request without a per-request delay is not held individually.
assert manager.get_request_delay(Request("http://example.com/b")) == 0.0
@coroutine_test
async def test_delay_scope(self):
manager = _manager(
{"THROTTLING_DEBUG": True, "RANDOMIZE_DOWNLOAD_DELAY": False}
)
manager = _manager({"THROTTLER_DEBUG": True, "RANDOMIZE_DOWNLOAD_DELAY": False})
request = Request("http://example.com/a")
await manager.get_scopes(request)
assert manager.is_ready(request) is True
@ -804,7 +802,7 @@ class TestThrottlingManagerReadiness:
@coroutine_test
async def test_reserve_blocks_on_concurrency(self):
manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}}})
manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}}})
first = Request("http://example.com/1")
second = Request("http://example.com/2")
await manager.get_scopes(first)
@ -820,7 +818,7 @@ class TestThrottlingManagerReadiness:
async def test_acquire_noop_when_reserved(self):
manager = _manager(
{
"THROTTLING_SCOPES": {"example.com": {"delay": 100.0}},
"THROTTLER_SCOPES": {"example.com": {"delay": 100.0}},
"RANDOMIZE_DOWNLOAD_DELAY": False,
}
)
@ -835,7 +833,7 @@ class TestThrottlingManagerReadiness:
@coroutine_test
async def test_get_scope_load(self):
manager = _manager({"THROTTLING_SCOPES": {"example.com": {"concurrency": 4}}})
manager = _manager({"THROTTLER_SCOPES": {"example.com": {"concurrency": 4}}})
assert manager.get_scope_load("example.com") == 0.0
request = Request("http://example.com/1")
await manager.get_scopes(request)
@ -937,11 +935,11 @@ class TestScopeHelpers:
update_scope_backoff({"a": 1.0}, "a", delay=1.0)
class TestThrottlingManagerEdges:
class TestThrottlerEdges:
@coroutine_test
async def test_acquire_without_scopes(self):
manager = _manager()
request = Request("http://example.com/a", meta={"throttling_scopes": []})
request = Request("http://example.com/a", meta={"throttler_scopes": []})
# No scopes resolve, so acquire() returns without reserving anything.
await manager.acquire(request)
assert request not in manager._reserved
@ -956,9 +954,9 @@ class TestThrottlingManagerEdges:
async def test_acquire_logs_and_waits_for_delay(self):
manager = _manager(
{
"THROTTLING_SCOPES": {"example.com": {"delay": 0.02}},
"THROTTLER_SCOPES": {"example.com": {"delay": 0.02}},
"RANDOMIZE_DOWNLOAD_DELAY": False,
"THROTTLING_DEBUG": True,
"THROTTLER_DEBUG": True,
}
)
r1 = Request("http://example.com/1")
@ -974,8 +972,8 @@ class TestThrottlingManagerEdges:
manager = _manager(
{
"THROTTLING_SCOPES": {"example.com": {"concurrency": 1}},
"THROTTLING_DEBUG": True,
"THROTTLER_SCOPES": {"example.com": {"concurrency": 1}},
"THROTTLER_DEBUG": True,
}
)
r1 = Request("http://example.com/1")
@ -988,10 +986,10 @@ class TestThrottlingManagerEdges:
@coroutine_test
async def test_delay_request(self):
manager = _manager({"THROTTLING_DEBUG": True})
request = Request("http://example.com/a", meta={"throttling_delay": 0.01})
manager = _manager({"THROTTLER_DEBUG": True})
request = Request("http://example.com/a", meta={"delay": 0.01})
await manager._delay_request(request)
assert request.meta["_throttling_delayed"] is True
assert request.meta["_throttler_delayed"] is True
# A second call is a no-op (the request was already delayed).
await manager._delay_request(request)
@ -1000,9 +998,9 @@ class TestThrottlingManagerEdges:
# Same as above but with debug logging off, so the delay is applied
# without emitting the debug message.
manager = _manager()
request = Request("http://example.com/a", meta={"throttling_delay": 0.01})
request = Request("http://example.com/a", meta={"delay": 0.01})
await manager._delay_request(request)
assert request.meta["_throttling_delayed"] is True
assert request.meta["_throttler_delayed"] is True
@coroutine_test
async def test_wait_for_slot_discards_unfired_events(self):
@ -1028,14 +1026,14 @@ class TestThrottlingManagerEdges:
assert manager._get_scope_manager("example.com")._backoff_level == 1
def test_apply_backoff_debug_logging(self, caplog):
manager = _manager({"THROTTLING_DEBUG": True})
with caplog.at_level(logging.DEBUG, logger="scrapy.throttling"):
manager = _manager({"THROTTLER_DEBUG": True})
with caplog.at_level(logging.DEBUG, logger="scrapy.throttler"):
manager._apply_backoff("example.com")
assert "Backoff for scope" in caplog.text
assert manager._get_scope_manager("example.com")._backoff_level == 1
def test_on_robots_parsed_disabled(self):
manager = _manager({"THROTTLING_ROBOTSTXT_OBEY": False})
manager = _manager({"THROTTLER_ROBOTSTXT_OBEY": False})
# With obeying disabled, the handler returns without touching any scope.
manager._on_robots_parsed(_FakeRobotParser(5.0), Request("http://example.com"))
assert not manager._scope_managers
@ -1044,21 +1042,21 @@ class TestThrottlingManagerEdges:
manager = _manager(
{
"ROBOTSTXT_OBEY": True,
"THROTTLING_SCOPES": {"example.com": {"delay": 0.5}},
"THROTTLER_SCOPES": {"example.com": {"delay": 0.5}},
}
)
with caplog.at_level(logging.WARNING, logger="scrapy.throttling"):
with caplog.at_level(logging.WARNING, logger="scrapy.throttler"):
manager.apply_robots_crawl_delay("example.com", 3.0)
assert "Crawl-delay" in caplog.text
def test_apply_robots_crawl_delay_debug_logging(self, caplog):
manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLING_DEBUG": True})
with caplog.at_level(logging.DEBUG, logger="scrapy.throttling"):
manager = _manager({"ROBOTSTXT_OBEY": True, "THROTTLER_DEBUG": True})
with caplog.at_level(logging.DEBUG, logger="scrapy.throttler"):
manager.apply_robots_crawl_delay("example.com", 3.0)
assert "Crawl-delay" in caplog.text
def test_maybe_evict_disabled(self):
manager = _manager({"THROTTLING_SCOPE_MAX_IDLE": 0})
manager = _manager({"THROTTLER_SCOPE_MAX_IDLE": 0})
scope = manager._get_scope_manager("example.com")
scope.record_sent(now=0.0)
scope.record_done(now=0.0)
@ -1067,7 +1065,7 @@ class TestThrottlingManagerEdges:
assert "example.com" in manager._scope_managers
class TestThrottlingScopeManagerEdges:
class TestThrottlerScopeManagerEdges:
def test_jitter_as_range(self):
scope = _scope_manager(
config={"id": "x", "backoff": {"jitter": [0.0, 0.0]}, "min_delay": 1.0}
@ -1134,7 +1132,7 @@ class TestThrottlingScopeManagerEdges:
assert scope.is_idle(now=0.0, max_idle=1.0) is True
class TestThrottlingIntegration:
class TestThrottlerIntegration:
@coroutine_test
async def test_backoff_recorded_on_429(self, mockserver):
crawler = get_crawler(SimpleSpider, {"RETRY_ENABLED": False})
@ -1144,21 +1142,21 @@ class TestThrottlingIntegration:
throttler = crawler.throttler
assert throttler is not None
managers = throttler._scope_managers
assert managers, "no throttling scope was created"
assert managers, "no throttler scope was created"
assert any(manager._backoff_level >= 1 for manager in managers.values())
@coroutine_test
async def test_backoff_recorded_on_download_error(self, mockserver):
crawler = get_crawler(SimpleSpider, {"RETRY_ENABLED": False})
# A dropped connection raises a DownloadFailedError, which the engine
# routes through the throttling manager before re-raising.
# routes through the throttler before re-raising.
await crawler.crawl_async(
mockserver.url("/drop?abort=1"), mockserver=mockserver
)
throttler = crawler.throttler
assert throttler is not None
managers = throttler._scope_managers
assert managers, "no throttling scope was created"
assert managers, "no throttler scope was created"
assert any(manager._backoff_level >= 1 for manager in managers.values())
@coroutine_test