From dc3ebb6cf76daa1953418af5aae3b83ffc12d02a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= <adrian@chaves.io>
Date: Thu, 7 Nov 2024 16:38:48 +0100
Subject: [PATCH] Refactor the docs

---
 docs/topics/autothrottle.rst | 57 ++++++++++++++++++++++--------------
 1 file changed, 35 insertions(+), 22 deletions(-)

diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rst
index cfd6440f2..5bd72fa15 100644
--- a/docs/topics/autothrottle.rst
+++ b/docs/topics/autothrottle.rst
@@ -21,9 +21,14 @@ Design goals
 How it works
 ============
 
-AutoThrottle extension adjusts download delays dynamically to make spider send
-:setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent requests on average
-to each remote website.
+Scrapy allows defining the concurrency and delay of different download slots,
+e.g. through the :setting:`DOWNLOAD_SLOTS` setting. By default requests are
+assigned to slots based on their URL domain, although it is possible to
+customize the download slot of any request.
+
+The AutoThrottle extension adjusts the delay of each download slot dynamically,
+to make your spider send :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent
+requests on average to each remote website.
 
 It uses download latency to compute the delays. The main idea is the
 following: if a server needs ``latency`` seconds to respond, a client
@@ -47,25 +52,6 @@ effect, but there are some important differences:
 
 AutoThrottle doesn't have these issues.
 
-.. reqmeta:: autothrottle_dont_adjust_delay
-
-Disabling the throttling of a request
-=====================================
-
-To disable AutoThrottle for a specific request, set the
-``autothrottle_dont_adjust_delay`` request metadata key to ``True``:
-
-.. code-block:: python
-
-    from scrapy import Request
-
-    Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})
-
-Note, however, that AutoThrottle still determines the starting delay of every
-slot by setting the ``download_delay`` attribute on the running spider. You
-might want to set a custom value for the ``delay`` attribute of the slot, e.g.
-using :setting:`DOWNLOAD_SLOTS`.
-
 Throttling algorithm
 ====================
 
@@ -99,6 +85,33 @@ callback, for example, and unable to attend downloads. However, these latencies
 should still give a reasonable estimate of how busy Scrapy (and ultimately, the
 server) is, and this extension builds on that premise.
 
+.. reqmeta:: autothrottle_dont_adjust_delay
+
+Prevent specific requests from triggering slot delay adjustments
+================================================================
+
+AutoThrottle adjusts the delay of download slots based on the latencies of
+responses that belong to that download slot. The only exceptions are non-200
+responses, which are only taken into account to increase that delay, but
+ignored if they would decrease that delay.
+
+You can also set the ``autothrottle_dont_adjust_delay`` request metadata key to
+``True`` in any request to prevent its response latency from impacting the
+delay of its download slot:
+
+.. code-block:: python
+
+    from scrapy import Request
+
+    Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})
+
+Note, however, that AutoThrottle still determines the starting delay of every
+download slot by setting the ``download_delay`` attribute on the running
+spider. If you want AutoThrottle not to impact a download slot at all, in
+addition to setting this meta key in all requests that use that download slot,
+you might want to set a custom value for the ``delay`` attribute of that
+download slot, e.g. using :setting:`DOWNLOAD_SLOTS`.
+
 Settings
 ========