Merge cf95286a4c into e28e56aa61

docs/topics/downloader-middleware.rst

@@ -829,6 +829,19 @@ redirect. For example, :class:`RedirectMiddleware` indicates the triggering
 response status code as an integer, while :class:`MetaRefreshMiddleware`
 always uses the ``'meta refresh'`` string as reason.
 
+.. reqmeta:: redirect_times
+
+The number of redirects that a request has gone through so far can be found in
+the ``redirect_times`` :attr:`Request.meta <scrapy.Request.meta>` key.
+
+.. reqmeta:: redirect_ttl
+
+The number of redirects that a request may still go through is tracked in the
+``redirect_ttl`` :attr:`Request.meta <scrapy.Request.meta>` key, which is
+initialized from :setting:`REDIRECT_MAX_TIMES` and decreased on every redirect.
+Set it on a request to allow fewer redirects for that request;
+:setting:`REDIRECT_MAX_TIMES` still applies as an upper bound.
+
 The :class:`RedirectMiddleware` can be configured through the following
 settings (see the settings documentation for more info):
 
@@ -951,6 +964,18 @@ RetryMiddleware
 If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
 set to True, the request will be ignored by this middleware.
 
+.. reqmeta:: retry_times
+
+The number of times that a request has been retried so far can be found in the
+``retry_times`` :attr:`Request.meta <scrapy.Request.meta>` key. Since it counts
+against :setting:`RETRY_TIMES`, copying it into a follow-up request lowers the
+number of retries allowed for that request.
+
+.. reqmeta:: priority_adjust
+
+The ``priority_adjust`` :attr:`Request.meta <scrapy.Request.meta>` key
+overrides :setting:`RETRY_PRIORITY_ADJUST` for a request.
+
 To retry requests from a spider callback, you can use the
 :func:`get_retry_request` function:
 
@@ -1055,6 +1080,8 @@ Adjust retry request priority relative to original request:
 - a positive priority adjust means higher priority.
 - **a negative priority adjust (default) means lower priority.**
 
+See also: :reqmeta:`priority_adjust`.
+
 
 .. _topics-dlmw-robots:
 

docs/topics/request-response.rst

@@ -161,7 +161,7 @@ Request objects
 
         A common use of request metadata is to define request-specific
         parameters for Scrapy components (extensions, middlewares, etc.). For
-        example, if you set ``dont_retry`` to ``True``,
+        example, if you set :reqmeta:`dont_retry` to ``True``,
         :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` will never
         retry that request, even if it fails. See :ref:`topics-request-meta`.
 
@@ -169,15 +169,15 @@ Request objects
         example, to keep request state information relevant to your component.
         For example,
         :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses the
-        ``retry_times`` metadata key to keep track of how many times a request
-        has been retried so far.
+        :reqmeta:`retry_times` metadata key to keep track of how many times a
+        request has been retried so far.
 
         Copying all the metadata of a previous request into a new, follow-up
         request in a spider callback is a bad practice, because request
         metadata may include metadata set by Scrapy components that is not
         meant to be copied into other requests. For example, copying the
-        ``retry_times`` metadata key into follow-up requests can lower the
-        amount of retries allowed for those follow-up requests.
+        :reqmeta:`retry_times` metadata key into follow-up requests can lower
+        the amount of retries allowed for those follow-up requests.
 
         You should only copy all request metadata from one request to another
         if the new request is meant to replace the old request, as is often the
@@ -828,6 +828,7 @@ Those are:
 * :reqmeta:`autothrottle_dont_adjust_delay`
 * :reqmeta:`bindaddress`
 * :reqmeta:`cookiejar`
+* :reqmeta:`depth`
 * :reqmeta:`dont_cache`
 * :reqmeta:`dont_merge_cookies`
 * :reqmeta:`dont_obey_robotstxt`
@@ -837,10 +838,12 @@ Those are:
 * :reqmeta:`download_latency`
 * :reqmeta:`download_maxsize`
 * :reqmeta:`download_slot`
-* :reqmeta:`download_warnsize`
 * :reqmeta:`download_timeout`
-* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
-* ``ftp_user`` (See :setting:`FTP_USER` for more info)
+* :reqmeta:`download_warnsize`
+* :reqmeta:`ftp_local_filename`
+* :reqmeta:`ftp_passive`
+* :reqmeta:`ftp_password`
+* :reqmeta:`ftp_user`
 * :reqmeta:`give_up_log_level`
 * :reqmeta:`handle_httpstatus_all`
 * :reqmeta:`handle_httpstatus_list`
@@ -848,13 +851,22 @@ Those are:
 * :reqmeta:`http_pass`
 * :reqmeta:`http_user`
 * :reqmeta:`is_start_request`
+* :reqmeta:`link_text`
 * :reqmeta:`max_retry_times`
+* :reqmeta:`priority_adjust`
 * :reqmeta:`proxy`
 * :reqmeta:`redirect_reasons`
+* :reqmeta:`redirect_times`
+* :reqmeta:`redirect_ttl`
 * :reqmeta:`redirect_urls`
 * :reqmeta:`referrer_policy`
+* :reqmeta:`retry_times`
+* :reqmeta:`rule`
 * :reqmeta:`verbatim_url`
 
+Scrapy components also use meta keys whose name starts with an underscore, such
+as ``_auth_proxy``. Those are internal, and may change or disappear at any time.
+
 .. reqmeta:: bindaddress
 
 bindaddress
@@ -921,6 +933,15 @@ download_fail_on_dataloss
 Whether or not to fail on broken responses. See:
 :setting:`DOWNLOAD_FAIL_ON_DATALOSS`.
 
+.. reqmeta:: ftp_local_filename
+
+ftp_local_filename
+------------------
+
+Path, as :class:`bytes`, of the file where to write the response body of an
+``ftp://`` request. If set, :attr:`Response.body <scrapy.http.Response.body>`
+holds this path instead of the file contents.
+
 .. reqmeta:: give_up_log_level
 
 give_up_log_level

docs/topics/settings.rst

@@ -1081,6 +1081,7 @@ handler (without replacement), place this in your ``settings.py``:
     :ref:`security-local-resources`
 
 
+.. reqmeta:: download_slot
 .. setting:: DOWNLOAD_SLOTS
 
 DOWNLOAD_SLOTS
@@ -1105,6 +1106,14 @@ Allows to define concurrency/delay parameters on per slot (domain) basis:
     -   :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency``
     -   :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay``
 
+Requests are assigned to a slot based on their URL domain. To assign a request
+to a specific slot instead, set the name of the slot as the ``download_slot``
+:attr:`Request.meta <scrapy.Request.meta>` key. Once a request is assigned to a
+slot, that key holds the name of the slot.
+
+Since that key is kept on redirects, a redirected request stays in the slot of
+the request it comes from, even when it points to a different domain.
+
 
 .. setting:: DOWNLOAD_TIMEOUT
 
@@ -1433,6 +1442,7 @@ non-default value in :ref:`per-spider settings <spider-settings>`.
 
 .. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
 
+.. reqmeta:: ftp_passive
 .. setting:: FTP_PASSIVE_MODE
 
 FTP_PASSIVE_MODE
@@ -1440,7 +1450,8 @@ FTP_PASSIVE_MODE
 
 Default: ``True``
 
-Whether or not to use passive mode when initiating FTP transfers.
+Whether or not to use passive mode when initiating FTP transfers, unless there
+is an ``"ftp_passive"`` key in ``Request`` meta.
 
 .. note::
 

docs/topics/spider-middleware.rst

@@ -228,6 +228,8 @@ DepthMiddleware
 .. module:: scrapy.spidermiddlewares.depth
    :synopsis: Depth Spider Middleware
 
+.. reqmeta:: depth
+
 .. autoclass:: DepthMiddleware
 
 HttpErrorMiddleware

docs/topics/spiders.rst

@@ -491,6 +491,15 @@ CrawlSpider
        described below. If multiple rules match the same link, the first one
        will be used, according to the order they're defined in this attribute.
 
+   .. reqmeta:: rule
+
+   Requests generated from :attr:`rules` carry the index of the matching rule
+   within :attr:`rules` in their ``rule``
+   :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs
+   that key to dispatch the response to the right rule, so copying it into a
+   request generated by a different rule sends the response to the wrong
+   callback.
+
    This spider also exposes an overridable method:
 
    .. method:: parse_start_url(response, **kwargs)
@@ -504,6 +513,8 @@ CrawlSpider
 Crawling rules
 ~~~~~~~~~~~~~~
 
+.. reqmeta:: link_text
+
 .. autoclass:: Rule
 
    ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which

scrapy/http/cookies.py

@@ -154,14 +154,11 @@ class WrappedRequest:
         return urlparse_cached(self.request).scheme
 
     def is_unverifiable(self) -> bool:
-        """Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965.
-
-        It defaults to False. An unverifiable request is one whose URL the user did not have the
-        option to approve. For example, if the request is for an image in an
-        HTML document, and the user had no option to approve the automatic
-        fetching of the image, this should be true.
+        """Return ``False``, as Scrapy does not track whether the user had the
+        option to approve the URL of a request, which is what makes a request
+        unverifiable as defined by :rfc:`2965`.
         """
-        return cast("bool", self.request.meta.get("is_unverifiable", False))
+        return False
 
     @property
     def full_url(self) -> str:

tests/test_http_cookies.py

@@ -79,11 +79,6 @@ class TestWrappedRequest:
         assert not self.wrapped.is_unverifiable()
         assert not self.wrapped.unverifiable
 
-    def test_is_unverifiable2(self):
-        self.request.meta["is_unverifiable"] = True
-        assert self.wrapped.is_unverifiable()
-        assert self.wrapped.unverifiable
-
     def test_get_origin_req_host(self):
         assert self.wrapped.origin_req_host == "www.example.com"