* Cover 2.17.0 in the release notes up to dd10cb8.
* Fix a copypasted issue number.
* Address minor issues
* Minor fixes
* Cover additional deprecations introduced while improving test coverage
* versionadded/versionchanged improvements
* Add a versionadded to the give_up_log_level docstring
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.
Closes#6120
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
* new 'allow_offsite' parameter in OffsiteMiddleware
* document deprecated dont_filter flag in OffsiteMiddleware
* avoid deprecating dont_filter in OffsiteMiddleware
* Copy the code to the downloader mw.
* Add tests for allow_offsite in the downloader mw.
* Mark allow_offsite with reqmeta.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
* Fix the display name of documented fingerprinter class methods
* Initial draft for the Scrapy 2.7 release notes
* Update VERSION and PREVIOUS_VERSION references
* Clarify the restrictions lifted for item field output names
* Fix the description of the BOM bug fix
* Fix the note about changes in MIME sniffing
* Fix typo
* Extend highlights
* Fyx typo
Andrey Rakhmatullin
Javier
Adrian
Labib Bin Salam
Adnan Awan
Adrián Chaves
Julian Hamze
Adham 3llam
Kevin Lloyd Bernal
Laerte Pereira
Jalil SA
Aryan
Aaron Smith
pankaj1707k
Mikhail Korobov
mattkohl-flex