diff --git a/docs/topics/components.rst b/docs/topics/components.rst index d122271fa..d7aa5642a 100644 --- a/docs/topics/components.rst +++ b/docs/topics/components.rst @@ -10,7 +10,6 @@ A Scrapy component is any class whose objects are built using That includes the classes that you may assign to the following settings: - :setting:`ADDONS` -- :setting:`TWISTED_DNS_RESOLVER` - :setting:`DOWNLOAD_HANDLERS` - :setting:`DOWNLOADER_MIDDLEWARES` - :setting:`DUPEFILTER_CLASS` @@ -27,6 +26,7 @@ That includes the classes that you may assign to the following settings: - :setting:`SPIDER_MIDDLEWARES` - :setting:`THROTTLING_MANAGER` - :setting:`THROTTLING_SCOPE_MANAGER` +- :setting:`TWISTED_DNS_RESOLVER` Third-party Scrapy components may also let you define additional Scrapy components, usually configurable through :ref:`settings `, to diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 83f28cf2a..fa353d126 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -174,22 +174,7 @@ For a list of the components enabled by default (and their orders) see the BackoffMiddleware ----------------- -.. module:: scrapy.downloadermiddlewares.backoff - :synopsis: Backoff Downloader Middleware - -.. class:: BackoffMiddleware - - This middleware feeds download outcomes into :ref:`throttling `: - for every response matching :setting:`BACKOFF_HTTP_CODES` and every download - exception matching :setting:`BACKOFF_EXCEPTIONS`, it makes the request's - :ref:`throttling scopes ` :ref:`back off `. - - It runs below - :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` so that it - observes rate-limiting responses (``429``, ``503``, …) before they are - turned into retries. - - See :ref:`throttling` for details. +.. autoclass:: scrapy.downloadermiddlewares.backoff.BackoffMiddleware .. _cookies-mw: diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 0d8441900..f4d2d0eaa 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -713,6 +713,7 @@ Those are: * :reqmeta:`dont_obey_robotstxt` * :reqmeta:`dont_redirect` * :reqmeta:`dont_retry` +* :reqmeta:`dont_throttle` * :reqmeta:`download_fail_on_dataloss` * :reqmeta:`download_latency` * :reqmeta:`download_maxsize` @@ -733,7 +734,6 @@ Those are: * :reqmeta:`redirect_urls` * :reqmeta:`referrer_policy` * :reqmeta:`throttling_delay` -* :reqmeta:`throttling_dont_track` * :reqmeta:`throttling_scopes` * :reqmeta:`verbatim_url` diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 964bd3ade..6d057626a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -861,7 +861,7 @@ Default: "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590, "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600, - "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 630, + "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 650, "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700, "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750, "scrapy.downloadermiddlewares.stats.DownloaderStats": 850, @@ -2039,7 +2039,6 @@ See :ref:`asyncio-without-reactor` for more information about this mode. .. versionadded:: 2.15.0 - .. setting:: TWISTED_REACTOR TWISTED_REACTOR diff --git a/docs/topics/throttling.rst b/docs/topics/throttling.rst index 2401c5321..50f6ca5a5 100644 --- a/docs/topics/throttling.rst +++ b/docs/topics/throttling.rst @@ -61,7 +61,6 @@ When configuring these settings, note that: more complex domain grouping strategies, see :ref:`alternative-domain-throttling`. - .. setting:: THROTTLING_SCOPES .. _per-domain-throttling: @@ -89,7 +88,6 @@ the :ref:`tutorial `, so that they are crawled faster while the Additional keys like ``"jitter"`` and ``"backoff"`` can be used here and are covered later on. - .. _backoff: Backoff @@ -417,7 +415,7 @@ Without a higher priority, a backlog of requests ahead of it in a FIFO queue could keep it waiting well past the configured delay; a higher priority puts it at the front of the queue, so it goes out right after its delay. -.. reqmeta:: throttling_dont_track +.. reqmeta:: dont_throttle Excluding a request from throttling state ----------------------------------------- @@ -425,12 +423,12 @@ Excluding a request from throttling state Some requests (authentication flows, one-off API calls, file downloads) should not influence throttling state even if they get a :setting:`BACKOFF_HTTP_CODES` response or raise a :setting:`BACKOFF_EXCEPTIONS` exception. Set the -``throttling_dont_track`` request metadata key to ``True`` to process such a +:reqmeta:`dont_throttle` request metadata key to ``True`` to process such a request normally without letting its outcome trigger :ref:`backoff `: .. code-block:: python - Request("https://example.com/login", meta={"throttling_dont_track": True}) + Request("https://example.com/login", meta={"dont_throttle": True}) .. _throttling-scopes: @@ -1080,47 +1078,6 @@ Additional settings long-running crawls. Set to ``0`` to never evict. Scopes in active backoff are never evicted. -.. _autothrottle-migration: - -Migrating from AutoThrottle -=========================== - -The ``AutoThrottle`` extension is deprecated in favor of the throttling and -:ref:`backoff ` system described here, which is always active and does -not need to be enabled. - -Setting ``AUTOTHROTTLE_ENABLED`` to ``True`` still works but logs a -deprecation warning. To migrate, drop the ``AUTOTHROTTLE_*`` settings and use -the following equivalents: - -.. list-table:: - :header-rows: 1 - - * - AutoThrottle - - Throttling - * - ``AUTOTHROTTLE_ENABLED = True`` - - No equivalent; throttling is always active. - * - ``AUTOTHROTTLE_START_DELAY`` - - :setting:`DOWNLOAD_DELAY` - * - ``AUTOTHROTTLE_MAX_DELAY`` - - :setting:`BACKOFF_MAX_DELAY` - * - ``AUTOTHROTTLE_TARGET_CONCURRENCY`` - - :ref:`rampup ` (``"rampup": True``) - * - ``AUTOTHROTTLE_DEBUG = True`` - - :setting:`THROTTLING_DEBUG` ``= True`` - * - ``autothrottle_dont_adjust_delay`` (request meta) - - :reqmeta:`throttling_dont_track` (request meta) - -AutoThrottle adjusted the delay of each download slot based on response -latency. The new system does not measure latency; instead, it reacts to -explicit rate-limit signals (:setting:`BACKOFF_HTTP_CODES`, -:setting:`BACKOFF_EXCEPTIONS`, :ref:`Retry-After / RateLimit-Reset -`) and, with :ref:`rampup `, probes for the -fastest rate a scope tolerates. If you specifically need latency-based control, -implement a custom :ref:`throttling scope manager -`. - - .. _throttling-api: API diff --git a/scrapy/downloadermiddlewares/backoff.py b/scrapy/downloadermiddlewares/backoff.py index 173ef1e62..da6f2a9f1 100644 --- a/scrapy/downloadermiddlewares/backoff.py +++ b/scrapy/downloadermiddlewares/backoff.py @@ -69,9 +69,7 @@ class BackoffMiddleware: ` to back off the request's scopes through its :meth:`~scrapy.throttling.ThrottlingManagerProtocol.back_off` API. - It sits below :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` - in :setting:`DOWNLOADER_MIDDLEWARES_BASE` so it sees rate-limiting responses - (429, 503, …) before the retry middleware turns them into new requests. + See :ref:`throttling` for details. """ def __init__(self, crawler: Crawler): @@ -118,7 +116,7 @@ class BackoffMiddleware: if ( response.status not in self._http_codes or "cached" in response.flags - or request.meta.get("throttling_dont_track") + or request.meta.get("dont_throttle") ): return response matched = [ @@ -139,7 +137,7 @@ class BackoffMiddleware: exception: Exception, spider: scrapy.Spider | None = None, ) -> None: - if request.meta.get("throttling_dont_track") or not isinstance( + if request.meta.get("dont_throttle") or not isinstance( exception, self._exceptions ): return diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 7df8ece5e..a363f2ec0 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -360,7 +360,7 @@ DOWNLOADER_MIDDLEWARES_BASE = { "scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware": 580, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 590, "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": 600, - "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 630, + "scrapy.downloadermiddlewares.backoff.BackoffMiddleware": 650, "scrapy.downloadermiddlewares.cookies.CookiesMiddleware": 700, "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750, "scrapy.downloadermiddlewares.stats.DownloaderStats": 850, diff --git a/tests/test_throttling.py b/tests/test_throttling.py index ddc14ff9b..72cabaa0c 100644 --- a/tests/test_throttling.py +++ b/tests/test_throttling.py @@ -275,7 +275,7 @@ class TestThrottlingManager: @coroutine_test async def test_response_backoff_dont_track(self): manager = _manager() - response = _response(status=429, meta={"throttling_dont_track": True}) + response = _response(status=429, meta={"dont_throttle": True}) assert await manager.get_response_backoff(response) is None @pytest.mark.parametrize( @@ -295,7 +295,7 @@ class TestThrottlingManager: @coroutine_test async def test_exception_backoff_dont_track(self): manager = _manager() - request = Request("http://example.com", meta={"throttling_dont_track": True}) + request = Request("http://example.com", meta={"dont_throttle": True}) assert ( await manager.get_exception_backoff(request, DownloadTimeoutError()) is None )