* Cover 2.17.0 in the release notes up to dd10cb8.
* Fix a copypasted issue number.
* Address minor issues
* Minor fixes
* Cover additional deprecations introduced while improving test coverage
* versionadded/versionchanged improvements
* Add a versionadded to the give_up_log_level docstring
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.
Closes#6120
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
* Allow configuring the log level of the retry give-up message
The "Gave up retrying ..." message was always logged at ERROR, which
inflates the log_count/ERROR stat even when giving up on a request is
expected (e.g. broad crawls hitting dead hosts).
Add a RETRY_GIVE_UP_LOG_LEVEL setting, a give_up_log_level argument to
get_retry_request(), and a give_up_log_level request meta key to override
it per request. The value accepts a level name ("WARNING") or number
(logging.WARNING). The default ("ERROR") preserves the previous behaviour.
Fixes#5297, fixes#4622
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address Adrian's review feedback: simplify and reorganize docs
- Move give_up_log_level reqmeta section before max_retry_times (alphabetical order)
- Simplify give_up_log_level section in request-response.rst (brief, links to setting)
- Simplify RETRY_GIVE_UP_LOG_LEVEL setting docs in downloader-middleware.rst
- Change 'When initialized' to 'When set' in max_retry_times section
- Docs now follow pattern of linking to complementary setting/meta key rather than duplicating information
Per Adrian's feedback: keep docs concise and cross-link setting ↔ meta key
* Address Adrian's code review feedback on RETRY_GIVE_UP_LOG_LEVEL feature
- Fix alphabetical ordering of give_up_log_level in request-response.rst
- Remove circular references: change 'see X for details' to 'see also X' pattern
- Simplify docstring for give_up_log_level parameter (4 lines → 2 lines)
- Update test domains from www.scrapytest.org to example.com
* Apply suggestion from @AdrianAtZyte
* Minor changes
* Fix example formatting.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrian <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* docs: switch `scrapy.Item` examples to dataclasses
* make serializer doc generic
* use modern type hints
* docs/spiders: switch TestItem consumer snippets to attribute access
Since the TestItem migration to @dataclass, the existing
item["id"] = ... assignments would raise TypeError on copy-paste.
Switch to item.id = ... to match the new dataclass declaration.
The snippets sit under .. skip: next so docs-tests still pass either
way, but the change keeps the examples runnable for readers.
* Remove support for sync process_spider_output().
* Slight check fix.
* Don't expect exceptions from calling process_spider_output().
* Remove dead code.
* Fix test_deprecated_mw_spider_arg() to run all methods.
* Fix typing.
* Typos.
* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective
* Fix indentation issue
* Minor rewordings
* older versions → lower versions
* Update news.rst
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Python 3.14 (alpha3) to CI.
* Disable mitmproxy on 3.14 for now.
* 3.14.0-alpha.4.
* 3.14.0-alpha.5
* 3.14.0-beta2.
* 3.14 release.
* Fix test_non_pickable_object.
* Fix handling of file:/path feed URIs.
* Better mocking of streams for TextTestResult.
* Do not use .php in test_file_path() as it's now a known extension.
* Fix the URL in TestFeedExporterSignals.
* Fix typing.
* Bump more envs to 3.14.
* Silence pylint.
* Fix another test for .php handling change.
* Remove test_install_asyncio_reactor.
* More bumps to 3.14.
* Revert docs-tests to use 3.13.
* Debug options for Windows.
* Re-enable xdist.
* Revert Windows PYTEST_ADDOPTS.
* Silence loop policy deprecation warnings.
* Restore a lost pylint suppression.
* Update asyncio_enabled_reactor_same_loop.py to new Twisted.
* Fix RobotFileParser tests for Python 3.14.5.
* Run mitmproxy-based tests for every handler and improve them.
* Fixes for H2DownloadHandler.
* Revert "Fixes for H2DownloadHandler."
This reverts commit bcdbd097cd.
* Add TestHttpsProxy for HTTP11DownloadHandler.
* Use the configured context factory in ScrapyProxyAgent.
* Reword.
* Reword.
* pragma: no cover for H2DownloadHandler proxy code.
* Raise an exception for HTTPS proxies instead of using HTTP.
* Remove non-working proxy support and explicitly forbid HTTP.
* pragma: no cover
* Add a docs note about HTTPS proxies.
* Rename ScrapyProxyAgent.
* Cover 2.15.0 in the release notes (up to 31bf7c3).
* Replace VERSION.
* Document unified download handler exceptions.
* Updates.
* Cover 2.15.0 in the release notes (up to 0c6ccf5).
* Cover 2.15.0 in the release notes (up to b68f267).
* Run linkcheck and fix issues.
* Cleanup unused link targets.
* Cleanup.
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
* Add security warning about pickle files in JOBDIR
The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Make JOBDIR security warning more concise and accurate
Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
* Initial release notes for 2.14.0, up to ed63fa9.
* Cover 2.14 in the release notes up to 393d715.
* Cover 2.14 in the release notes up to eb49647.
* Cover 2.14 in the release notes up to 426aafd.
* Cover 2.14 in the release notes up to 1e8de24.
* Cover 2.14 in the release notes up to 5a7e132.
* Bump sphinx-lint.
* Finalize the 2.14.0 release notes.
* Drop more of the old versionadded directives.
* Address feedback.
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
Adrian
Andrey Rakhmatullin
Javier
smellslikeml
Labib Bin Salam
Adnan Awan
Omkar Kabde
SpiliosDmk
Olivia Choi
Abhinav W
Adrián Chaves
Sanjay M
NI283105
Kailesh
Ahmed Arshad
Julian Hamze
Yuval Elbar
Anant Murmu
Martin Schimandl
Michael
Adham 3llam
Thalison Fernandes
Sebastien Dionne
Felipe Benevolo
Laerte Pereira