WIP

docs/topics/components.rst

@@ -10,7 +10,6 @@ A Scrapy component is any class whose objects are built using
 That includes the classes that you may assign to the following settings:
 
 - :setting:`ADDONS`
-- :setting:`TWISTED_DNS_RESOLVER`
 - :setting:`DOWNLOAD_HANDLERS`
 - :setting:`DOWNLOADER_MIDDLEWARES`
 - :setting:`DUPEFILTER_CLASS`
@@ -27,6 +26,7 @@ That includes the classes that you may assign to the following settings:
 - :setting:`SPIDER_MIDDLEWARES`
 - :setting:`THROTTLING_MANAGER`
 - :setting:`THROTTLING_SCOPE_MANAGER`
+- :setting:`TWISTED_DNS_RESOLVER`
 
 Third-party Scrapy components may also let you define additional Scrapy
 components, usually configurable through :ref:`settings <topics-settings>`, to

docs/topics/downloader-middleware.rst

@@ -174,22 +174,7 @@ For a list of the components enabled by default (and their orders) see the
 BackoffMiddleware
 -----------------
 
-.. module:: scrapy.downloadermiddlewares.backoff
-   :synopsis: Backoff Downloader Middleware
-
-.. class:: BackoffMiddleware
-
-   This middleware feeds download outcomes into :ref:`throttling <throttling>`:
-   for every response matching :setting:`BACKOFF_HTTP_CODES` and every download
-   exception matching :setting:`BACKOFF_EXCEPTIONS`, it makes the request's
-   :ref:`throttling scopes <throttling-scopes>` :ref:`back off <backoff>`.
-
-   It runs below
-   :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` so that it
-   observes rate-limiting responses (``429``, ``503``, …) before they are
-   turned into retries.
-
-   See :ref:`throttling` for details.
+.. autoclass:: scrapy.downloadermiddlewares.backoff.BackoffMiddleware
 
 .. _cookies-mw:
 

docs/topics/request-response.rst

@@ -713,6 +713,7 @@ Those are:
 * :reqmeta:`dont_obey_robotstxt`
 * :reqmeta:`dont_redirect`
 * :reqmeta:`dont_retry`
+* :reqmeta:`dont_throttle`
 * :reqmeta:`download_fail_on_dataloss`
 * :reqmeta:`download_latency`
 * :reqmeta:`download_maxsize`
@@ -733,7 +734,6 @@ Those are:
 * :reqmeta:`redirect_urls`
 * :reqmeta:`referrer_policy`
 * :reqmeta:`throttling_delay`
-* :reqmeta:`throttling_dont_track`
 * :reqmeta:`throttling_scopes`
 * :reqmeta:`verbatim_url`
 

docs/topics/settings.rst

@@ -861,7 +861,7 @@ Default:
         "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,
         "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,
         "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,
-        "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 630,
+        "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 650,
         "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,
         "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
         "scrapy.downloadermiddlewares.stats.DownloaderStats": 850,
@@ -2039,7 +2039,6 @@ See :ref:`asyncio-without-reactor` for more information about this mode.
 
 .. versionadded:: 2.15.0
 
-
 .. setting:: TWISTED_REACTOR
 
 TWISTED_REACTOR

docs/topics/throttling.rst

@@ -61,7 +61,6 @@ When configuring these settings, note that:
     more complex domain grouping strategies, see
     :ref:`alternative-domain-throttling`.
 
-
 .. setting:: THROTTLING_SCOPES
 
 .. _per-domain-throttling:
@@ -89,7 +88,6 @@ the :ref:`tutorial <intro-tutorial>`, so that they are crawled faster while the
 Additional keys like ``"jitter"`` and ``"backoff"`` can be used here and are
 covered later on.
 
-
 .. _backoff:
 
 Backoff
@@ -417,7 +415,7 @@ Without a higher priority, a backlog of requests ahead of it in a FIFO queue
 could keep it waiting well past the configured delay; a higher priority puts it
 at the front of the queue, so it goes out right after its delay.
 
-.. reqmeta:: throttling_dont_track
+.. reqmeta:: dont_throttle
 
 Excluding a request from throttling state
 -----------------------------------------
@@ -425,12 +423,12 @@ Excluding a request from throttling state
 Some requests (authentication flows, one-off API calls, file downloads) should
 not influence throttling state even if they get a :setting:`BACKOFF_HTTP_CODES`
 response or raise a :setting:`BACKOFF_EXCEPTIONS` exception. Set the
-``throttling_dont_track`` request metadata key to ``True`` to process such a
+:reqmeta:`dont_throttle` request metadata key to ``True`` to process such a
 request normally without letting its outcome trigger :ref:`backoff <backoff>`:
 
 .. code-block:: python
 
-    Request("https://example.com/login", meta={"throttling_dont_track": True})
+    Request("https://example.com/login", meta={"dont_throttle": True})
 
 .. _throttling-scopes:
 
@@ -1080,47 +1078,6 @@ Additional settings
     long-running crawls. Set to ``0`` to never evict. Scopes in active backoff
     are never evicted.
 
-.. _autothrottle-migration:
-
-Migrating from AutoThrottle
-===========================
-
-The ``AutoThrottle`` extension is deprecated in favor of the throttling and
-:ref:`backoff <backoff>` system described here, which is always active and does
-not need to be enabled.
-
-Setting ``AUTOTHROTTLE_ENABLED`` to ``True`` still works but logs a
-deprecation warning. To migrate, drop the ``AUTOTHROTTLE_*`` settings and use
-the following equivalents:
-
-.. list-table::
-    :header-rows: 1
-
-    * - AutoThrottle
-      - Throttling
-    * - ``AUTOTHROTTLE_ENABLED = True``
-      - No equivalent; throttling is always active.
-    * - ``AUTOTHROTTLE_START_DELAY``
-      - :setting:`DOWNLOAD_DELAY`
-    * - ``AUTOTHROTTLE_MAX_DELAY``
-      - :setting:`BACKOFF_MAX_DELAY`
-    * - ``AUTOTHROTTLE_TARGET_CONCURRENCY``
-      - :ref:`rampup <rampup>` (``"rampup": True``)
-    * - ``AUTOTHROTTLE_DEBUG = True``
-      - :setting:`THROTTLING_DEBUG` ``= True``
-    * - ``autothrottle_dont_adjust_delay`` (request meta)
-      - :reqmeta:`throttling_dont_track` (request meta)
-
-AutoThrottle adjusted the delay of each download slot based on response
-latency. The new system does not measure latency; instead, it reacts to
-explicit rate-limit signals (:setting:`BACKOFF_HTTP_CODES`,
-:setting:`BACKOFF_EXCEPTIONS`, :ref:`Retry-After / RateLimit-Reset
-<rate-limiting-headers>`) and, with :ref:`rampup <rampup>`, probes for the
-fastest rate a scope tolerates. If you specifically need latency-based control,
-implement a custom :ref:`throttling scope manager
-<custom-throttling-scope-managers>`.
-
-
 .. _throttling-api:
 
 API

scrapy/downloadermiddlewares/backoff.py

@@ -69,9 +69,7 @@ class BackoffMiddleware:
     <throttling>` to back off the request's scopes through its
     :meth:`~scrapy.throttling.ThrottlingManagerProtocol.back_off` API.
 
-    It sits below :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`
-    in :setting:`DOWNLOADER_MIDDLEWARES_BASE` so it sees rate-limiting responses
-    (429, 503, …) before the retry middleware turns them into new requests.
+    See :ref:`throttling` for details.
     """
 
     def __init__(self, crawler: Crawler):
@@ -118,7 +116,7 @@ class BackoffMiddleware:
         if (
             response.status not in self._http_codes
             or "cached" in response.flags
-            or request.meta.get("throttling_dont_track")
+            or request.meta.get("dont_throttle")
         ):
             return response
         matched = [
@@ -139,7 +137,7 @@ class BackoffMiddleware:
         exception: Exception,
         spider: scrapy.Spider | None = None,
     ) -> None:
-        if request.meta.get("throttling_dont_track") or not isinstance(
+        if request.meta.get("dont_throttle") or not isinstance(
             exception, self._exceptions
         ):
             return

scrapy/settings/default_settings.py

@@ -360,7 +360,7 @@ DOWNLOADER_MIDDLEWARES_BASE = {
     "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580,
     "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590,
     "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600,
-    "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 630,
+    "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 650,
     "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700,
     "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
     "scrapy.downloadermiddlewares.stats.DownloaderStats": 850,

tests/test_throttling.py

@@ -275,7 +275,7 @@ class TestThrottlingManager:
     @coroutine_test
     async def test_response_backoff_dont_track(self):
         manager = _manager()
-        response = _response(status=429, meta={"throttling_dont_track": True})
+        response = _response(status=429, meta={"dont_throttle": True})
         assert await manager.get_response_backoff(response) is None
 
     @pytest.mark.parametrize(
@@ -295,7 +295,7 @@ class TestThrottlingManager:
     @coroutine_test
     async def test_exception_backoff_dont_track(self):
         manager = _manager()
-        request = Request("http://example.com", meta={"throttling_dont_track": True})
+        request = Request("http://example.com", meta={"dont_throttle": True})
         assert (
             await manager.get_exception_backoff(request, DownloadTimeoutError()) is None
         )