ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` ( #7388 )
...
* deprecate `walk_modules` in favour of `walk_modules_iter`
* remove deprecation line from docstring
* addressing review
2026-04-01 22:11:35 +05:00
0c6ccf50b3
Docs for the reactorless mode ( #7385 )
...
* Reactorless mode docs.
* Unset the ftp handler in the reactorless mode.
* Cleanup reactorless subprocess tests.
* Better handling of incompatible TWISTED_ENABLED cases.
* Improvements.
* Improve early error logging in test scripts.
* Address feedback.
* Address feedback.
2026-04-01 20:40:34 +05:00
eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES ( #7379 )
...
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.
* Deprecate BrowserLikeContextFactory.
* Add DOWNLOAD_VERIFY_CERTIFICATES.
* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
86a7ceaa9f
Dynamic loading for S3 HTTPS handler ( #7370 )
2026-03-30 10:12:43 +02:00
fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. ( #7366 )
2026-03-27 12:21:25 +01:00
5561aaec1d
fixing oversight from #7036 ( #7367 )
...
https://github.com/scrapy/scrapy/pull/7269#issuecomment-3958535816
2026-03-27 11:38:44 +01:00
2ce02d417a
TLS code modernization ( #7353 )
...
* Import some stuff from service_identity directly.
* Extract _log_tls().
* Sync validating IP addresses.
* Introduce _deps_compat.
* Deprecate getCertificateOptions().
* Deprecate ScrapyClientTLSOptions.
* Add pragma: no cover.
* verify=False and trustRoot=platformTrust() are the defaults.
* Improve docstrings for TLS classes.
* Deprecate AcceptableProtocolsContextFactory.
* Switch from DefaultOpenSSLContextFactory to CertificateOptions.
* Deprecate load_context_factory_from_settings().
* Remove the outdated warning about incompatible factories.
* Simplify _load_context_factory_from_settings().
* Make CertificateOptions and Context once per factory.
* pragma: no cover
2026-03-25 15:42:43 +05:00
03d105ac92
Add warnings to components not working without a reactor ( #7355 )
2026-03-25 09:21:08 +01:00
54a4c3af89
Deprecate the mail API ( #7263 )
2026-03-24 21:31:12 +01:00
bfe34492fa
asyncio alternative to deferToThread() ( #7349 )
2026-03-24 15:41:15 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
d42b23d78a
Await persist_file() in media pipelines. ( #7182 )
2026-03-24 10:52:56 +01:00
9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. ( #7276 )
2026-03-24 10:36:48 +01:00
939db88b04
Disable mypy allow_untyped_calls. ( #7331 )
2026-03-24 10:31:17 +01:00
c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers ( #7283 )
2026-03-19 09:16:40 +01:00
584d99af30
Split long test files ( #7329 )
2026-03-13 10:30:25 +01:00
9dfe449d13
Fix test issues with the no-reactor env
2026-03-12 16:46:32 +01:00
4e3df249f2
Merge branch '2.14'
2026-03-12 16:02:37 +01:00
8e28f938d2
Make test_no_warning_when_referer_middleware_present less brittle
2026-03-12 14:29:51 +01:00
886131c7b2
Run pre-commit
2026-03-12 13:54:00 +01:00
945b787a26
Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14
2026-03-12 13:37:24 +01:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
7010985e4f
Bump ruff, fix some of rules ( #7277 )
2026-02-23 11:48:38 +01:00
abd025f78e
Pass correct env to PopenSpawn() in tests. ( #7279 )
2026-02-23 11:34:56 +01:00
3fe89a211b
Fix open_spider exception handling to set exit code 1 ( #7255 )
2026-02-19 11:21:59 +01:00
ccfa052fa1
Enable in-process HTTP tests without a reactor. ( #7254 )
2026-02-13 19:08:06 +01:00
6e0a0e476a
Fix running tests with -n auto, enable it on CI ( #7257 )
...
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
09bd8f4231
Properly close ftplib.FTP(). ( #7256 )
2026-02-13 19:12:13 +05:00
0e1526ed30
Fix flaky test_download_with_proxy_https_timeout(). ( #7250 )
2026-02-13 09:45:09 +01:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
06fb87f7bb
Fix override behavior in getwithbase() issue #6912 ( #6993 )
...
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly
* Test actual import paths from default_settings
* Improvements
* Remove unused logger
* Run pre-commit
* Restore and improve duplicate key handling and warning
* type → object
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-07 00:17:55 +04:00
294ee051cc
Reducing memory footprint of `Request` object: __slots__ and lazy evaluation ( #7036 )
...
* reducing memory foorprint of `Request` object
* restore flow
* Test setters
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:15:53 +04:00
2e53d90e4c
Refactor HTTP download handlers ( #7238 )
2026-02-03 20:13:30 +01:00
11977afba5
Rename async test decorators. ( #7234 )
2026-02-02 20:39:43 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
54d8562fb0
Assorted test improvements ( #7232 )
...
* Await some missed deferreds.
* Simplify test_engine_stop_download_*.
* Fix the header value.
* Extend TestHttpWithCrawlerBase.
* Make test_tls_logging more universal.
* Add more tests for header handling.
* Clarify certificate and ip_address integration tests.
* Make test_download_with_maxsize_very_large_file() more universal.
* Fix test_coroutine_asyncio().
* Typo.
2026-02-02 09:37:39 +01:00
4e1faf883d
Foundations for the reactorless mode. ( #7199 )
...
* Foundations for the reactorless mode.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
2026-01-31 20:48:54 +05:00
3ec6ae05c1
Fix omitting repeated dataloss warnings in HTTP11DownloadHandler. ( #7222 )
2026-01-22 00:41:27 +05:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
ba3d7bc7a8
Remove the non-standard 307/308 handling, and align other aspects with the standard
2026-01-21 18:46:05 +01:00
9bae1ee21f
Lazy creation of Downloader._slot_gc_loop. ( #7210 )
2026-01-20 12:58:09 +01:00
842d0becf0
Rename test function
2026-01-20 12:15:25 +01:00
f7f18123eb
Remove body on 307/308 redirects and switch to GET on 301
2026-01-19 19:15:20 +01:00
b6e5c58ae7
Fix arbitrary callable execution from the Referrer-Policy response header
2026-01-19 16:44:03 +01:00
0f07b2e38c
Pass the spider arg to custom stat collectors {open,close}_spider(). ( #7213 )
2026-01-09 16:57:53 +01:00
d1bd8eb49f
Async API for download handlers. ( #7164 )
2025-12-31 19:03:16 +01:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
5a7e132486
Restore and deprecate the spider arg of Downloader._get_slot(). ( #7178 )
2025-12-15 11:42:24 +01:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
6ba6b032ad
Fix handling meta["download_slot"] == None. ( #7172 )
2025-12-11 09:46:51 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
483e059d59
Deprecate returning Deferreds from spider callbacks and errbacks. ( #7151 )
2025-12-09 15:03:32 +01:00
11073c8680
Deprecate spider attributes that can be replaced by settings, round 2 ( #7039 )
...
* Move duplicate code to utils
* move new function to end
* draft
* update docs
* Update tests
* Update test name
* rename test
* rollback signature
* leftover
* sort
* Rollback some changes
* Rollback download_delay warning
* Rollback
* fix checks
* Remove unused imports
* Add pragma: no cover.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-01 01:26:17 +05:00
1e8de24380
Merge pull request #7145 from Icorebleidd/master
...
Fix duplicate assertion in test_utils_deprecate
2025-11-28 01:16:02 +05:00
813fd9f1ac
Restore brotlicffi support ( #7160 )
...
* restore brotlicffi dependency
* pre-commit
2025-11-25 12:06:55 +05:00
14737e91ed
Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
e48a1bdde3
Speed up and refactor command tests ( #7118 )
2025-10-27 12:28:16 +01:00
5afc9b0221
Automatically transpose images using EXIF data ( #6975 )
...
* Automatically transpose images using EXIF data
* Added test
---------
Co-authored-by: Andrew Murray <radarhere@users.noreply.github.com>
2025-10-24 16:08:23 +05:00
e98c1644ce
Add unit tests that check download handlers behave like basic http clients ( #6946 ) ( #7059 )
2025-10-20 13:08:26 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
7b215c6578
UTF-8 BOM at the beginning of the file ignored ( #7095 )
2025-10-15 18:15:07 +04:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
10850e7d29
Fix extra stdout output in tests ( #7070 )
2025-10-06 11:19:34 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
dd0b071bcc
Deprecate sync methods of ItemPipelineManager/ITEM_PROCESSOR. ( #7043 )
2025-10-03 14:15:42 +02:00
d239fcf936
Fix get_meta_refresh to correctly resolve relative URLs using <base> tag ( #7047 )
2025-09-10 15:17:16 +02:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
c097921c44
Extract @_warn_spider_arg. ( #7033 )
...
* Extract @_warn_spider_arg.
* Also deprecate passing spider=None.
2025-09-01 20:17:13 +05:00
80beec41b5
Log when AsyncCrawlerProcess or CrawlerProcess is instantiated. ( #7034 )
2025-09-01 13:51:11 +02:00
00b2be0943
Add unit tests for the StatsMailer extension ( #7013 )
2025-09-01 11:43:57 +02:00
3ee4a52aa1
Raise NotConfigured when FILES_STORE or IMAGES_STORE is None or empty ( #6969 )
2025-09-01 11:39:39 +02:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
15655ca834
Improve cleanup in TestEngineCloseSpider. ( #7012 )
2025-08-18 20:00:23 +05:00
3c546bdb82
Refactor chainDeferred() usages ( #7008 )
2025-08-16 09:35:23 +02:00
d3e15a10cf
Add ItemPipelineManager.process_item_async() and ensure_awaitable(). ( #7005 )
...
* Add ItemPipelineManager.process_item_async() and ensure_awaitable().
* Wording.
2025-08-12 19:57:37 +05:00
57f539d7a5
Deprecate Crawler.stop(). ( #6999 )
2025-08-11 08:32:55 +02:00
a0b766f9e1
More *_async APIs ( #6997 )
...
* Add ExecutionEngine.close_spider_async().
* Add Scraper.{open,close}_spider_async().
* Fix double engine stopping via Crawler.stop().
* Add ExecutionEngine.stop_async().
* Add versionadded to new async APIs.
* Add ExecutionEngine.close_async().
2025-08-08 22:15:41 +05:00
d27d0a4ed9
Deprecate setting overrides through spider attributes ( #6994 )
2025-08-07 21:26:10 +02:00
a3daa3612e
Rewrite ExecutionEngine.close_spider() to inlineCallbacks. ( #6986 )
2025-08-06 08:01:55 +02:00
baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses ( #6984 )
...
* Add a crawler argument to MiddlewareManager.__init__().
* Deprecate the spider argument in some MiddlewareManager subclasses.
* Remove the spider argument from ExecutionEngine.open_spider_async().
* Remove the spider argument from Scraper.open_spider().
* Fix the Scraper.close_spider() type hint.
* Remove the spider argument from Downloader internals.
* Deprecate the spider arg of download_func.
* Deprecate the spider arg of Downloader.fetch().
* Deprecate the spider arg of ItemPipelineManager.process_item().
* Fix pylint.
* Tests for MiddlewareManager spider code.
* Deprecated spider arg handling for custom pipeline managers.
* Add a test for the deprecated spider arg in a custom downloader.
* Remove TODO.
* Fix pylint.
2025-08-05 12:21:08 +05:00
32f2aede82
Deprecate non-async counterparts of new async APIs. ( #6979 )
...
* Deprecate non-async counterparts of new async APIs.
* Deprecate Scraper.call_spider().
* Add _schedule_coro().
2025-08-01 00:58:48 +05:00
605e7669a5
Improve the assumption in test_queue_push_pop_priorities(). ( #6974 )
2025-07-28 20:37:45 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list ( #6949 )
2025-07-19 11:54:35 +02:00
e225d0dea4
Ban more imports that import twisted.internet.reactor. ( #6947 )
2025-07-10 21:19:50 +05:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
14eace5d8f
Increase the command timeout in test_short_timeout(). ( #6945 )
2025-07-06 01:56:54 +05:00
4279f2837c
Enable the flake8-builtins rules. ( #6942 )
2025-07-05 21:43:18 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
Albert Eduardovich N.
Andrey Rakhmatullin
Kailesh
Julian Hamze
Adrian Chaves
pierreeurope
Ryotaro
Adrián Chaves
Thalison Fernandes
Laerte Pereira
Leonardo Tozzo
Rui Xi
Andrew Murray
Wes Galbraith
Vasiliy Kiryanov
Ashar Khan
M Ikram Ullah Khan
NavpreetSSidhu
Felipe Benevolo