3a88cd0e2b
2.15.2 ( #7468 )
...
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467 )
* Release notes for 2.15.2
2026-04-28 15:26:42 +02:00
528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 ( #7467 )
2026-04-28 12:17:12 +02:00
4e25686b20
Merge branch '2.15'
2026-04-23 21:08:38 +05:00
f3c5a6e75f
Bump version: 2.15.0 → 2.15.1
2026-04-23 19:57:44 +05:00
416a454dc5
Release notes for 2.15.1. ( #7456 )
2026-04-23 19:51:00 +05:00
3561748280
sphinx-scrapy: 0.7.1 → 0.8.3 ( #7454 )
2026-04-23 13:20:30 +05:00
41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 ( #7454 )
2026-04-23 10:03:09 +02:00
508367664f
Fix missing load_object import and grammar in docs ( #7417 )
...
Co-authored-by: NI <Nidotexe@gmail.com>
2026-04-09 15:31:01 +02:00
47e25fbbb5
Bump version: 2.14.2 → 2.15.0
2026-04-09 17:00:40 +05:00
1432455d35
Add a missing release notes change.
2026-04-09 16:59:18 +05:00
7e881ce2d7
Release notes for 2.15.0 ( #7373 )
...
* Cover 2.15.0 in the release notes (up to 31bf7c30c6ccf5b68f267
2026-04-09 16:56:30 +05:00
b9be5ce053
Reactorless shell and other small shell improvements ( #7395 )
...
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
2026-04-08 18:24:51 +05:00
13a014d2e6
Response now uses less memory ( #7374 )
2026-04-08 14:25:11 +02:00
a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER ( #7361 )
2026-04-08 13:58:35 +02:00
830eaeab5d
docs: fix typos in addons.rst ( #7408 )
...
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
2026-04-08 09:38:09 +02:00
010faf1722
Use sphinx-scrapy 0.7.1 ( #7406 )
2026-04-06 15:29:59 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. ( #7394 )
2026-04-02 10:20:54 +02:00
0c6ccf50b3
Docs for the reactorless mode ( #7385 )
...
* Reactorless mode docs.
* Unset the ftp handler in the reactorless mode.
* Cleanup reactorless subprocess tests.
* Better handling of incompatible TWISTED_ENABLED cases.
* Improvements.
* Improve early error logging in test scripts.
* Address feedback.
* Address feedback.
2026-04-01 20:40:34 +05:00
fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. ( #7387 )
...
* Move TLS logging from _ScrapyClientTLSOptions to handlers.
* Update the docs.
* Fix typing.
* Update new docs.
* Add the comment back.
2026-04-01 16:02:55 +05:00
eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES ( #7379 )
...
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.
* Deprecate BrowserLikeContextFactory.
* Add DOWNLOAD_VERIFY_CERTIFICATES.
* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
c4c0555ccf
Enable parallel execution in the other internal Sphinx plugin
2026-03-30 18:01:29 +02:00
299993b62a
Fix the docs build
2026-03-30 17:46:55 +02:00
31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed ( #7368 )
...
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
2026-03-27 21:06:55 +05:00
54a4c3af89
Deprecate the mail API ( #7263 )
2026-03-24 21:31:12 +01:00
d42b23d78a
Await persist_file() in media pipelines. ( #7182 )
2026-03-24 10:52:56 +01:00
c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers ( #7283 )
2026-03-19 09:16:40 +01:00
4e3df249f2
Merge branch '2.14'
2026-03-12 16:02:37 +01:00
498b4fc1a4
Bump version: 2.14.1 → 2.14.2
2026-03-12 15:45:16 +01:00
378bb68039
Proofread the release notes
2026-03-12 15:44:54 +01:00
945b787a26
Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14
2026-03-12 13:37:24 +01:00
da1a6b7ebc
Add security warning about pickle files in JOBDIR ( #7274 )
...
* Add security warning about pickle files in JOBDIR
The job directory contains files serialized with pickle (spider.state,
request queues), which can execute arbitrary code when loaded. Add a
warning so users know to treat JOBDIR with the same trust level as
their project code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Make JOBDIR security warning more concise and accurate
Simplify the warning to focus on treating the job directory
as trusted, without mentioning specific serialization details.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 19:36:26 +05:00
2c3ecbff71
fix: missing self argument in ScreenshotPipeline constructor ( #7248 )
...
added missing self argument in __init__ method constructor
2026-02-13 09:42:26 +01:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
66fe5de139
Add Jobdir documentation ( #5260 )
...
* Add Jobdir documentation
* Fix link format
* remove doubtful comments
* some more edits
* Keep the information in jobs.rst an example, and provide details in the reference docs of the corresponding components
* Minor edits
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-06 22:33:02 +04:00
6a42bc6450
Item docs: mention Pydantic support ( #6966 )
...
* Add Pydantic support documentation
* Fix Sphinx references in Pydantic section for non-Sphinx docs
* Minor reformatting
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:18:01 +04:00
8974580e43
Reword the release note entry to consider the 301 redirect fix a security bug fix
2026-02-03 20:38:11 +01:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
ba3d7bc7a8
Remove the non-standard 307/308 handling, and align other aspects with the standard
2026-01-21 18:46:05 +01:00
a39545195e
allow to override → allow overriding
2026-01-20 12:20:37 +01:00
1b9c8b55da
not → now
2026-01-20 12:13:44 +01:00
2651d48f20
Cover the 301 behavior change in the release notes
2026-01-19 19:25:00 +01:00
99d5d58e20
Add a release notes entry
2026-01-19 19:19:02 +01:00
6f03f3250b
Add versionadded
2026-01-19 16:46:53 +01:00
b6e5c58ae7
Fix arbitrary callable execution from the Referrer-Policy response header
2026-01-19 16:44:03 +01:00
0b9d8da09d
Bump version: 2.14.0 → 2.14.1
2026-01-12 16:25:43 -03:00
c9fbf6c599
Release notes for 2.14.1 ( #7217 )
...
* Release notes for 2.14.1
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update docs/news.rst
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
* Update docs/news.rst
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* Update message about github action change
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-01-12 16:23:51 -03:00
3ac1192f35
Bump version: 2.13.4 → 2.14.0
2026-01-05 15:06:13 +05:00
7bef98b4f1
Release notes for 2.14.0 ( #7035 )
...
* Initial release notes for 2.14.0, up to ed63fa9393d715eb49647426aafd1e8de245a7e132
2026-01-05 14:51:05 +05:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
4cb0144b39
Merge tag '2.13.4'
...
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:47:54 +05:00
2f62ab532d
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:30:37 +05:00
31a9c03c24
Release notes for 2.13.4. ( #7144 )
...
* Release notes for 2.13.4.
* Formatting.
2025-11-17 16:30:04 +05:00
d091256c58
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-11-14 15:39:21 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
2d073a9c0d
Fix typo in logging documentation ( #7109 )
2025-10-20 15:52:46 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
f2531808f3
docs: Add Edit on GitHub ( #7058 )
2025-09-28 13:16:47 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
d9ef0350d8
Merge remote-tracking branch 'origin/2.13'
2025-07-02 20:46:02 +05:00
155a504f24
Bump version: 2.13.2 → 2.13.3
2025-07-02 20:39:46 +05:00
cf465bf644
Release notes for 2.13.3. ( #6934 )
...
* Release notes for 2.13.3.
* Address feedback.
2025-07-02 20:36:01 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
03fe7a6424
Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:48 +05:00
2c1c10e923
Remove deprecated offsite spider middleware. ( #6926 )
...
* Remove deprecated offsite spider middleware.
* Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:20 +05:00
ba1cedee1b
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:13:54 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
1473f4d347
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-06-28 18:04:06 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
843ad1afb1
Replace Black with Ruff in contributing.rst. ( #6903 )
2025-06-23 22:25:56 +05:00
712e965dbd
Replace Black with Ruff in contributing.rst. ( #6903 )
2025-06-23 17:56:48 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
0390176ecd
Merge branch '2.13'
2025-06-09 15:54:35 +05:00
c6740604a4
Bump version: 2.13.1 → 2.13.2
2025-06-09 15:21:06 +05:00
7400868ad5
Release notes for 2.13.2. ( #6868 )
...
* Release notes for 2.13.2.
* Update release notes.
2025-06-09 15:19:02 +05:00
405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. ( #6866 )
2025-06-06 12:59:49 +02:00
d99234a33f
Install the reactor explicitly in CrawlerRunner examples. ( #6865 )
2025-06-06 11:54:21 +02:00
b8cd079014
Shorten the README and remove broken links to scrapy.org ( #6833 )
2025-06-06 12:44:02 +05:00
105c0afb6e
Feature the new logo in the README ( #6831 )
2025-06-06 12:43:56 +05:00
Adrian
Andrey Rakhmatullin
NI283105
Albert Eduardovich N.
Kailesh
Ahmed Arshad
Julian Hamze
Yuval Elbar
Anant Murmu
Martin Schimandl
Michael
Adrián Chaves
Eugenio Lacuesta
Eugenio Lacuesta
Adham 3llam
Thalison Fernandes
sstegmanbsd
Sebastien Dionne
Laerte Pereira
Felipe Benevolo