RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917)

This commit is contained in:
Adrian 2026-08-09 11:08:06 +02:00 committed by GitHub
parent 5b4888a0b1
commit 81d12c6eb8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/topics/settings.rst
View File

@ -658,6 +658,11 @@ The default headers used for Scrapy HTTP Requests. They're populated in the
:class:`Request.cookies <scrapy.Request>` parameter. This is a known
current limitation that is being worked on.
.. caution:: A ``Referer`` header defined here only reaches requests for which
:class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` does not set
one, such as start requests. To send it on every request, set
:setting:`REFERRER_POLICY` to ``"no-referrer"``.
.. setting:: DEPTH_LIMIT
DEPTH_LIMIT