diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index dec9904d2..25da37669 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -829,6 +829,19 @@ redirect. For example, :class:`RedirectMiddleware` indicates the triggering response status code as an integer, while :class:`MetaRefreshMiddleware` always uses the ``'meta refresh'`` string as reason. +.. reqmeta:: redirect_times + +The number of redirects that a request has gone through so far can be found in +the ``redirect_times`` :attr:`Request.meta ` key. + +.. reqmeta:: redirect_ttl + +The number of redirects that a request may still go through is tracked in the +``redirect_ttl`` :attr:`Request.meta ` key, which is +initialized from :setting:`REDIRECT_MAX_TIMES` and decreased on every redirect. +Set it on a request to allow fewer redirects for that request; +:setting:`REDIRECT_MAX_TIMES` still applies as an upper bound. + The :class:`RedirectMiddleware` can be configured through the following settings (see the settings documentation for more info): @@ -951,6 +964,18 @@ RetryMiddleware If :attr:`Request.meta ` has ``dont_retry`` key set to True, the request will be ignored by this middleware. +.. reqmeta:: retry_times + +The number of times that a request has been retried so far can be found in the +``retry_times`` :attr:`Request.meta ` key. Since it counts +against :setting:`RETRY_TIMES`, copying it into a follow-up request lowers the +number of retries allowed for that request. + +.. reqmeta:: priority_adjust + +The ``priority_adjust`` :attr:`Request.meta ` key +overrides :setting:`RETRY_PRIORITY_ADJUST` for a request. + To retry requests from a spider callback, you can use the :func:`get_retry_request` function: @@ -1055,6 +1080,8 @@ Adjust retry request priority relative to original request: - a positive priority adjust means higher priority. - **a negative priority adjust (default) means lower priority.** +See also: :reqmeta:`priority_adjust`. + .. _topics-dlmw-robots: diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 1d97e39b6..5e97cf64f 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -161,7 +161,7 @@ Request objects A common use of request metadata is to define request-specific parameters for Scrapy components (extensions, middlewares, etc.). For - example, if you set ``dont_retry`` to ``True``, + example, if you set :reqmeta:`dont_retry` to ``True``, :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` will never retry that request, even if it fails. See :ref:`topics-request-meta`. @@ -169,15 +169,15 @@ Request objects example, to keep request state information relevant to your component. For example, :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses the - ``retry_times`` metadata key to keep track of how many times a request - has been retried so far. + :reqmeta:`retry_times` metadata key to keep track of how many times a + request has been retried so far. Copying all the metadata of a previous request into a new, follow-up request in a spider callback is a bad practice, because request metadata may include metadata set by Scrapy components that is not meant to be copied into other requests. For example, copying the - ``retry_times`` metadata key into follow-up requests can lower the - amount of retries allowed for those follow-up requests. + :reqmeta:`retry_times` metadata key into follow-up requests can lower + the amount of retries allowed for those follow-up requests. You should only copy all request metadata from one request to another if the new request is meant to replace the old request, as is often the @@ -828,6 +828,7 @@ Those are: * :reqmeta:`autothrottle_dont_adjust_delay` * :reqmeta:`bindaddress` * :reqmeta:`cookiejar` +* :reqmeta:`depth` * :reqmeta:`dont_cache` * :reqmeta:`dont_merge_cookies` * :reqmeta:`dont_obey_robotstxt` @@ -837,10 +838,12 @@ Those are: * :reqmeta:`download_latency` * :reqmeta:`download_maxsize` * :reqmeta:`download_slot` -* :reqmeta:`download_warnsize` * :reqmeta:`download_timeout` -* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info) -* ``ftp_user`` (See :setting:`FTP_USER` for more info) +* :reqmeta:`download_warnsize` +* :reqmeta:`ftp_local_filename` +* :reqmeta:`ftp_passive` +* :reqmeta:`ftp_password` +* :reqmeta:`ftp_user` * :reqmeta:`give_up_log_level` * :reqmeta:`handle_httpstatus_all` * :reqmeta:`handle_httpstatus_list` @@ -848,13 +851,22 @@ Those are: * :reqmeta:`http_pass` * :reqmeta:`http_user` * :reqmeta:`is_start_request` +* :reqmeta:`link_text` * :reqmeta:`max_retry_times` +* :reqmeta:`priority_adjust` * :reqmeta:`proxy` * :reqmeta:`redirect_reasons` +* :reqmeta:`redirect_times` +* :reqmeta:`redirect_ttl` * :reqmeta:`redirect_urls` * :reqmeta:`referrer_policy` +* :reqmeta:`retry_times` +* :reqmeta:`rule` * :reqmeta:`verbatim_url` +Scrapy components also use meta keys whose name starts with an underscore, such +as ``_auth_proxy``. Those are internal, and may change or disappear at any time. + .. reqmeta:: bindaddress bindaddress @@ -921,6 +933,15 @@ download_fail_on_dataloss Whether or not to fail on broken responses. See: :setting:`DOWNLOAD_FAIL_ON_DATALOSS`. +.. reqmeta:: ftp_local_filename + +ftp_local_filename +------------------ + +Path, as :class:`bytes`, of the file where to write the response body of an +``ftp://`` request. If set, :attr:`Response.body ` +holds this path instead of the file contents. + .. reqmeta:: give_up_log_level give_up_log_level diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index e58815fad..086d84ba0 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1081,6 +1081,7 @@ handler (without replacement), place this in your ``settings.py``: :ref:`security-local-resources` +.. reqmeta:: download_slot .. setting:: DOWNLOAD_SLOTS DOWNLOAD_SLOTS @@ -1105,6 +1106,14 @@ Allows to define concurrency/delay parameters on per slot (domain) basis: - :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency`` - :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay`` +Requests are assigned to a slot based on their URL domain. To assign a request +to a specific slot instead, set the name of the slot as the ``download_slot`` +:attr:`Request.meta ` key. Once a request is assigned to a +slot, that key holds the name of the slot. + +Since that key is kept on redirects, a redirected request stays in the slot of +the request it comes from, even when it points to a different domain. + .. setting:: DOWNLOAD_TIMEOUT @@ -1433,6 +1442,7 @@ non-default value in :ref:`per-spider settings `. .. note:: This is a :ref:`pre-crawler setting `. +.. reqmeta:: ftp_passive .. setting:: FTP_PASSIVE_MODE FTP_PASSIVE_MODE @@ -1440,7 +1450,8 @@ FTP_PASSIVE_MODE Default: ``True`` -Whether or not to use passive mode when initiating FTP transfers. +Whether or not to use passive mode when initiating FTP transfers, unless there +is an ``"ftp_passive"`` key in ``Request`` meta. .. note:: diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 1c9ee0c77..6caad6149 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -228,6 +228,8 @@ DepthMiddleware .. module:: scrapy.spidermiddlewares.depth :synopsis: Depth Spider Middleware +.. reqmeta:: depth + .. autoclass:: DepthMiddleware HttpErrorMiddleware diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 95c80d5dc..bd8f697ed 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -491,6 +491,15 @@ CrawlSpider described below. If multiple rules match the same link, the first one will be used, according to the order they're defined in this attribute. + .. reqmeta:: rule + + Requests generated from :attr:`rules` carry the index of the matching rule + within :attr:`rules` in their ``rule`` + :attr:`Request.meta ` key. :class:`CrawlSpider` needs + that key to dispatch the response to the right rule, so copying it into a + request generated by a different rule sends the response to the wrong + callback. + This spider also exposes an overridable method: .. method:: parse_start_url(response, **kwargs) @@ -504,6 +513,8 @@ CrawlSpider Crawling rules ~~~~~~~~~~~~~~ +.. reqmeta:: link_text + .. autoclass:: Rule ``link_extractor`` is a :ref:`Link Extractor ` object which diff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.py index 555d930e6..8008ca9d2 100644 --- a/scrapy/http/cookies.py +++ b/scrapy/http/cookies.py @@ -154,14 +154,11 @@ class WrappedRequest: return urlparse_cached(self.request).scheme def is_unverifiable(self) -> bool: - """Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965. - - It defaults to False. An unverifiable request is one whose URL the user did not have the - option to approve. For example, if the request is for an image in an - HTML document, and the user had no option to approve the automatic - fetching of the image, this should be true. + """Return ``False``, as Scrapy does not track whether the user had the + option to approve the URL of a request, which is what makes a request + unverifiable as defined by :rfc:`2965`. """ - return cast("bool", self.request.meta.get("is_unverifiable", False)) + return False @property def full_url(self) -> str: diff --git a/tests/test_http_cookies.py b/tests/test_http_cookies.py index 5d4d5a2bd..8840172a5 100644 --- a/tests/test_http_cookies.py +++ b/tests/test_http_cookies.py @@ -79,11 +79,6 @@ class TestWrappedRequest: assert not self.wrapped.is_unverifiable() assert not self.wrapped.unverifiable - def test_is_unverifiable2(self): - self.request.meta["is_unverifiable"] = True - assert self.wrapped.is_unverifiable() - assert self.wrapped.unverifiable - def test_get_origin_req_host(self): assert self.wrapped.origin_req_host == "www.example.com"