* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* docs: switch `scrapy.Item` examples to dataclasses
* make serializer doc generic
* use modern type hints
* docs/spiders: switch TestItem consumer snippets to attribute access
Since the TestItem migration to @dataclass, the existing
item["id"] = ... assignments would raise TypeError on copy-paste.
Switch to item.id = ... to match the new dataclass declaration.
The snippets sit under .. skip: next so docs-tests still pass either
way, but the change keeps the examples runnable for readers.
* Remove support for sync process_spider_output().
* Slight check fix.
* Don't expect exceptions from calling process_spider_output().
* Remove dead code.
* Fix test_deprecated_mw_spider_arg() to run all methods.
* Fix typing.
* Typos.
* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective
* Fix indentation issue
* Minor rewordings
* older versions → lower versions
* Update news.rst
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Python 3.14 (alpha3) to CI.
* Disable mitmproxy on 3.14 for now.
* 3.14.0-alpha.4.
* 3.14.0-alpha.5
* 3.14.0-beta2.
* 3.14 release.
* Fix test_non_pickable_object.
* Fix handling of file:/path feed URIs.
* Better mocking of streams for TextTestResult.
* Do not use .php in test_file_path() as it's now a known extension.
* Fix the URL in TestFeedExporterSignals.
* Fix typing.
* Bump more envs to 3.14.
* Silence pylint.
* Fix another test for .php handling change.
* Remove test_install_asyncio_reactor.
* More bumps to 3.14.
* Revert docs-tests to use 3.13.
* Debug options for Windows.
* Re-enable xdist.
* Revert Windows PYTEST_ADDOPTS.
* Silence loop policy deprecation warnings.
* Restore a lost pylint suppression.
* Update asyncio_enabled_reactor_same_loop.py to new Twisted.
* Fix RobotFileParser tests for Python 3.14.5.
* Run mitmproxy-based tests for every handler and improve them.
* Fixes for H2DownloadHandler.
* Revert "Fixes for H2DownloadHandler."
This reverts commit bcdbd097cd.
* Add TestHttpsProxy for HTTP11DownloadHandler.
* Use the configured context factory in ScrapyProxyAgent.
* Reword.
* Reword.
* pragma: no cover for H2DownloadHandler proxy code.
* Raise an exception for HTTPS proxies instead of using HTTP.
* Remove non-working proxy support and explicitly forbid HTTP.
* pragma: no cover
* Add a docs note about HTTPS proxies.
* Rename ScrapyProxyAgent.
* Cover 2.15.0 in the release notes (up to 31bf7c3).
* Replace VERSION.
* Document unified download handler exceptions.
* Updates.
* Cover 2.15.0 in the release notes (up to 0c6ccf5).
* Cover 2.15.0 in the release notes (up to b68f267).
* Run linkcheck and fix issues.
* Cleanup unused link targets.
* Cleanup.
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
Andrey Rakhmatullin
Adnan Awan
Omkar Kabde
Adrian
Fardin Alizadeh
SpiliosDmk
Olivia Choi
Abhinav W
Adrián Chaves
Sanjay M
NI283105
Albert Eduardovich N.
Kailesh
Ahmed Arshad