d2290c35c2
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes #5297, fixes #4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
|
||
|---|---|---|
| .. | ||
| _ext | ||
| _static | ||
| _templates | ||
| _tests | ||
| intro | ||
| topics | ||
| utils | ||
| Makefile | ||
| README.rst | ||
| conf.py | ||
| conftest.py | ||
| contributing.rst | ||
| faq.rst | ||
| index.rst | ||
| news.rst | ||
| requirements.in | ||
| requirements.txt | ||
| versioning.rst | ||
README.rst
| orphan: |
|---|
Scrapy documentation quick start guide
This file provides a quick guide on how to compile the Scrapy documentation.
Setup the environment
To compile the documentation you need Sphinx Python library. To install it and all its dependencies run the following command from this dir
pip install -r requirements.txt
Compile the documentation
To compile the documentation (to classic HTML output) run the following command from this dir:
make html
Documentation will be generated (in HTML format) inside the build/html dir.
View the documentation
To view the documentation run the following command:
make htmlview
This command will fire up your default browser and open the main page of your (previously generated) HTML documentation.
Start over
To clean up all generated documentation files and start from scratch run:
make clean
Keep in mind that this command won't touch any documentation source files.
Recreating documentation on the fly
There is a way to recreate the doc automatically when you make changes, you need to install watchdog (pip install watchdog) and then use:
make watch
Alternative method using tox
To compile the documentation to HTML run the following command:
tox -e docs
Documentation will be generated inside the docs/_build/all dir.