mirror of https://github.com/scrapy/scrapy.git
d2290c35c2
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes #5297, fixes #4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
|
||
|---|---|---|
| .. | ||
| commands | ||
| contracts | ||
| core | ||
| downloadermiddlewares | ||
| extensions | ||
| http | ||
| linkextractors | ||
| loader | ||
| pipelines | ||
| selector | ||
| settings | ||
| spidermiddlewares | ||
| spiders | ||
| templates | ||
| utils | ||
| VERSION | ||
| __init__.py | ||
| __main__.py | ||
| addons.py | ||
| cmdline.py | ||
| crawler.py | ||
| dupefilters.py | ||
| exceptions.py | ||
| exporters.py | ||
| extension.py | ||
| interfaces.py | ||
| item.py | ||
| link.py | ||
| logformatter.py | ||
| mail.py | ||
| middleware.py | ||
| mime.types | ||
| pqueues.py | ||
| py.typed | ||
| resolver.py | ||
| responsetypes.py | ||
| robotstxt.py | ||
| shell.py | ||
| signalmanager.py | ||
| signals.py | ||
| spiderloader.py | ||
| squeues.py | ||
| statscollectors.py | ||