From d18085dc3efe348b9eaca0410d3e311576a70f03 Mon Sep 17 00:00:00 2001 From: Adrian Chaves Date: Wed, 24 Jun 2026 11:49:51 +0200 Subject: [PATCH] Update docs --- docs/topics/settings.rst | 1 + docs/topics/throttling.rst | 120 +++++++++++++++++++++++-------------- 2 files changed, 77 insertions(+), 44 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 405805133..1c875d4d3 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1917,6 +1917,7 @@ See :ref:`asyncio-without-reactor` for more information about this mode. .. versionadded:: 2.15.0 + .. setting:: TWISTED_REACTOR TWISTED_REACTOR diff --git a/docs/topics/throttling.rst b/docs/topics/throttling.rst index f376b135d..9b4337089 100644 --- a/docs/topics/throttling.rst +++ b/docs/topics/throttling.rst @@ -4,6 +4,8 @@ Throttling ========== +.. versionadded:: VERSION + Sending too many requests too quickly can `overwhelm websites`_. :ref:`Throttling ` and :ref:`backoff ` aim to prevent that. @@ -45,13 +47,17 @@ The main throttling :ref:`settings ` are: - .. setting:: DOWNLOAD_DELAY_PER_SLOT - :setting:`DOWNLOAD_DELAY_PER_SLOT` (default: ``1.0``) + :setting:`DOWNLOAD_DELAY_PER_SLOT` (default: ``None``) - Minimum seconds between requests in the same slot. + Minimum seconds to wait between two consecutive requests sent to the same + download slot. Unlike :setting:`DOWNLOAD_DELAY`, which applies per domain + (:ref:`throttling scope `), this delay is per slot. - If a slot sends a request and receives its response before this delay has - elapsed, it must wait before sending the next request. The wait time is - measured from when the previous request was sent. + When ``None`` (default), the per-slot delay falls back to + :setting:`DOWNLOAD_DELAY`, preserving the historical behavior where + :setting:`DOWNLOAD_DELAY` was enforced per slot. + + The wait time is measured from when the previous request was sent. For example, with ``CONCURRENT_REQUESTS_PER_DOMAIN = 2``, ``DOWNLOAD_DELAY = 0.3``, and ``DOWNLOAD_DELAY_PER_SLOT = 1.0``, sending 3 requests to the same domain @@ -79,13 +85,17 @@ When configuring these settings, note that: .. setting:: THROTTLING_SCOPES + .. _per-domain-throttling: Per-domain throttling ===================== -The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling behavior -for specific domains [1]_. +The :setting:`THROTTLING_SCOPES` setting allows you to customize throttling +behavior for specific domains [1]_. + +It is a dict that maps scope names to +:class:`~scrapy.throttling.ThrottlingScopeConfig` dicts. Its default value allows faster crawling of the testing website using during the :ref:`tutorial ` while maintaining conservative defaults @@ -118,19 +128,22 @@ The key settings are: :setting:`BACKOFF_HTTP_CODES` (default: ``[429, 502, 503, 504, 520, 521, 522, 523, 524]``) - HTTP status codes that trigger backoff. + HTTP response status codes that trigger backoff. - .. setting:: BACKOFF_DELAY_FACTOR :setting:`BACKOFF_DELAY_FACTOR` (default: ``2.0``) - Each backoff multiplies delay by this factor (2x, 4x, 8x, etc.). + Factor by which the delay of a scope is multiplied on each backoff step + (2×, 4×, 8×, etc.). - .. setting:: BACKOFF_MAX_DELAY :setting:`BACKOFF_MAX_DELAY` (default: ``300.0``) - Maximum delay cap to prevent excessively long waits. + Maximum delay, in seconds, that backoff can reach. Also caps + :ref:`Retry-After ` and :ref:`RateLimit-Reset + ` delays. .. _rampup: @@ -158,8 +171,9 @@ setting: :setting:`RAMPUP_BACKOFF_TARGET` (default: ``1``) - Target number of backoff responses per rampup window, indicating optimal - throughput. Can be a range like ``[1, 3]``. + Target number of backoff responses per :setting:`BACKOFF_WINDOW` that + :ref:`rampup ` aims for when probing the rate limit of a scope. + Can be a range like ``[1, 3]``. .. _retry-after: @@ -637,17 +651,12 @@ Additional settings Default: - .. code-block:: python + - :exc:`~scrapy.exceptions.DownloadFailedError` + - :exc:`~scrapy.exceptions.DownloadTimeoutError` + - :exc:`~scrapy.exceptions.ResponseDataLossError` - [ - "twisted.internet.defer.TimeoutError", - "twisted.internet.error.TimeoutError", - "twisted.internet.error.TCPTimedOutError", - "twisted.web.client.ResponseFailed", - ] - - Exception classes that trigger backoff. Strings are interpreted as import - paths. + List of exception classes that trigger backoff when raised while + downloading a request. Strings are interpreted as import paths. .. seealso:: :setting:`RETRY_EXCEPTIONS` @@ -655,46 +664,45 @@ Additional settings :setting:`BACKOFF_JITTER` (default: ``0.1``) + Random jitter applied to each backoff delay, as a fraction of the delay. + With the default value of ``0.1`` the delay is randomized by ±10%. Overrides :setting:`RANDOMIZE_DOWNLOAD_DELAY` during backoff. - .. setting:: BACKOFF_MIN_DELAY :setting:`BACKOFF_MIN_DELAY` (default: ``1.0``) - Minimum delay during :ref:`backoff `. + Delay, in seconds, applied on the first backoff step (and the minimum + delay during backoff). - .. setting:: BACKOFF_WINDOW :setting:`BACKOFF_WINDOW` (default: ``60.0``) - During :ref:`backoff `, after a non-backoff response is received, - do not take the next step in backoff reduction until this amount of time - has passed and no new backoff feedback has been received. - - The number of seconds that need to pass since the last non-backoff response - without any other for the backoff to move towards the original throttling - configuration. + Time window, in seconds, used by :ref:`backoff ` and + :ref:`rampup `. During backoff, a :ref:`throttling scope + ` must go this many seconds without a new backoff + trigger (an HTTP error code from :setting:`BACKOFF_HTTP_CODES` or an + exception from :setting:`BACKOFF_EXCEPTIONS`) before its delay decreases + by one :setting:`BACKOFF_DELAY_FACTOR` step toward the configured value. + A new trigger resets the countdown. - .. setting:: DELAYED_REQUESTS_WARN_THRESHOLD :setting:`DELAYED_REQUESTS_WARN_THRESHOLD` (default: ``500``) - While throttled, requests in the :ref:`scheduler ` remain - in the scheduler. + Number of requests held back by :ref:`throttling ` at which + Scrapy logs a warning, to help detect throttling configurations that hold + back more requests than expected. - However, requests sent with :meth:`engine.download() - ` bypass the scheduler. This - includes requests sent by some built-in :ref:`components - ` and :ref:`inline requests `. - - When such requests are throttled, they are paused and kept in memory, along + While throttled, requests in the :ref:`scheduler ` + remain in the scheduler. However, requests sent with :meth:`engine.download() + ` bypass the scheduler, + including requests sent by some built-in :ref:`components + ` and :ref:`inline requests `. When + such requests are throttled, they are paused and kept in memory, along with any run time context from the code that is sending them. If they - accumulate, they can become a memory issue that may require you to rethink - your throttling parameters or crawl strategy. - - :setting:`DELAYED_REQUESTS_WARN_THRESHOLD` defines a threshold for such - requests. The first time that this many such requests are being throttled - at the same time, a warning is issued. + accumulate, they can become a memory issue. - .. setting:: RANDOMIZE_DOWNLOAD_DELAY @@ -710,6 +718,30 @@ Additional settings If ``True``, ``0.5`` (i.e. ±50%) is used as the randomization factor. If ``False``, no randomization is applied. +- .. setting:: TARGET_RPM + + :setting:`TARGET_RPM` (default: ``None``) + + Target number of requests per minute *per domain*. It has no effect on its + own; it is read by :class:`~scrapy.addons.throttling.TargetRPMAddon`, which + must be enabled explicitly. + +- .. setting:: THROTTLING_DEBUG + + :setting:`THROTTLING_DEBUG` (default: ``False``) + + Whether to log :ref:`throttling ` decisions (per-scope delays, + backoff steps and recoveries) for debugging. + +- .. setting:: THROTTLING_SCOPE_MAX_IDLE + + :setting:`THROTTLING_SCOPE_MAX_IDLE` (default: ``3600.0``) + + Seconds of inactivity after which the state of a :ref:`throttling scope + ` is evicted from memory to bound memory usage on + long-running crawls. Set to ``0`` to never evict. Scopes in active backoff + are never evicted. + .. _throttling-api: