From 81d12c6eb895e2026372dbb54a20e58597c69c5f Mon Sep 17 00:00:00 2001 From: Adrian Date: Sun, 9 Aug 2026 11:08:06 +0200 Subject: [PATCH] Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) --- docs/topics/settings.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 65ee77258..793d65f35 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -658,6 +658,11 @@ The default headers used for Scrapy HTTP Requests. They're populated in the :class:`Request.cookies ` parameter. This is a known current limitation that is being worked on. +.. caution:: A ``Referer`` header defined here only reaches requests for which + :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` does not set + one, such as start requests. To send it on every request, set + :setting:`REFERRER_POLICY` to ``"no-referrer"``. + .. setting:: DEPTH_LIMIT DEPTH_LIMIT