mirror of https://github.com/scrapy/scrapy.git
d2290c35c2
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes #5297, fixes #4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
|
||
|---|---|---|
| .. | ||
| _images | ||
| addons.rst | ||
| api.rst | ||
| architecture.rst | ||
| asyncio.rst | ||
| autothrottle.rst | ||
| benchmarking.rst | ||
| broad-crawls.rst | ||
| commands.rst | ||
| components.rst | ||
| contracts.rst | ||
| coroutines.rst | ||
| debug.rst | ||
| deploy.rst | ||
| developer-tools.rst | ||
| djangoitem.rst | ||
| download-handlers.rst | ||
| downloader-middleware.rst | ||
| dynamic-content.rst | ||
| exceptions.rst | ||
| exporters.rst | ||
| extensions.rst | ||
| feed-exports.rst | ||
| item-pipeline.rst | ||
| items.rst | ||
| jobs.rst | ||
| leaks.rst | ||
| link-extractors.rst | ||
| loaders.rst | ||
| logging.rst | ||
| media-pipeline.rst | ||
| practices.rst | ||
| request-response.rst | ||
| scheduler.rst | ||
| scrapyd.rst | ||
| selectors.rst | ||
| settings.rst | ||
| shell.rst | ||
| signals.rst | ||
| spider-middleware.rst | ||
| spiders.rst | ||
| stats.rst | ||
| telnetconsole.rst | ||