Commit Graph

2537 Commits

Author SHA1 Message Date
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386)
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
Albert Eduardovich N. e3a8ff2b59
improve trackref (#7375)
* improve trackref

- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`

* tracing GC is no joke

* refine tests

* explain the tests

* add note for pypy in docs + ...

return empty tuple instead of list in `iter_all`
2026-04-03 15:15:30 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 510f09a961
Deprecate ScrapyClientContextFactory. (#7391) 2026-04-02 09:29:47 +02:00
Albert Eduardovich N. ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` (#7388)
* deprecate `walk_modules` in favour of `walk_modules_iter`

* remove deprecation line from docstring

* addressing review
2026-04-01 22:11:35 +05:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385)
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Kailesh 86a7ceaa9f
Dynamic loading for S3 HTTPS handler (#7370) 2026-03-30 10:12:43 +02:00
Andrey Rakhmatullin fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. (#7366) 2026-03-27 12:21:25 +01:00
Albert Eduardovich N. 5561aaec1d
fixing oversight from #7036 (#7367)
https://github.com/scrapy/scrapy/pull/7269#issuecomment-3958535816
2026-03-27 11:38:44 +01:00
Andrey Rakhmatullin 2ce02d417a
TLS code modernization (#7353)
* Import some stuff from service_identity directly.

* Extract _log_tls().

* Sync validating IP addresses.

* Introduce _deps_compat.

* Deprecate getCertificateOptions().

* Deprecate ScrapyClientTLSOptions.

* Add pragma: no cover.

* verify=False and trustRoot=platformTrust() are the defaults.

* Improve docstrings for TLS classes.

* Deprecate AcceptableProtocolsContextFactory.

* Switch from DefaultOpenSSLContextFactory to CertificateOptions.

* Deprecate load_context_factory_from_settings().

* Remove the outdated warning about incompatible factories.

* Simplify _load_context_factory_from_settings().

* Make CertificateOptions and Context once per factory.

* pragma: no cover
2026-03-25 15:42:43 +05:00
Andrey Rakhmatullin 03d105ac92
Add warnings to components not working without a reactor (#7355) 2026-03-25 09:21:08 +01:00
Kaileshwar R K 54a4c3af89
Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
Andrey Rakhmatullin bfe34492fa
asyncio alternative to deferToThread() (#7349) 2026-03-24 15:41:15 +01:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Andrey Rakhmatullin 9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. (#7276) 2026-03-24 10:36:48 +01:00
Andrey Rakhmatullin 939db88b04
Disable mypy allow_untyped_calls. (#7331) 2026-03-24 10:31:17 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Andrey Rakhmatullin 584d99af30
Split long test files (#7329) 2026-03-13 10:30:25 +01:00
Adrian Chaves 9dfe449d13 Fix test issues with the no-reactor env 2026-03-12 16:46:32 +01:00
Adrian Chaves 4e3df249f2 Merge branch '2.14' 2026-03-12 16:02:37 +01:00
Adrian Chaves 8e28f938d2 Make test_no_warning_when_referer_middleware_present less brittle 2026-03-12 14:29:51 +01:00
Adrian Chaves 886131c7b2 Run pre-commit 2026-03-12 13:54:00 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Andrey Rakhmatullin e02ad08672
Move to mypy --strict with exceptions. (#7300)
* Move to mypy --strict with exceptions.

* Fix typing-tests.
2026-03-02 15:47:23 +05:00
Andrey Rakhmatullin 7010985e4f
Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin abd025f78e
Pass correct env to PopenSpawn() in tests. (#7279) 2026-02-23 11:34:56 +01:00
pierreeurope 3fe89a211b
Fix open_spider exception handling to set exit code 1 (#7255) 2026-02-19 11:21:59 +01:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 6e0a0e476a
Fix running tests with -n auto, enable it on CI (#7257)
* Make test_start_deprecated_super() more robust.

* Work around the FTPFilesStore.FTP_* initialization.

* Enable xdist on CI.

* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
Andrey Rakhmatullin 09bd8f4231
Properly close ftplib.FTP(). (#7256) 2026-02-13 19:12:13 +05:00
Andrey Rakhmatullin 0e1526ed30
Fix flaky test_download_with_proxy_https_timeout(). (#7250) 2026-02-13 09:45:09 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Ryotaro 06fb87f7bb
Fix override behavior in getwithbase() issue #6912 (#6993)
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly

* Test actual import paths from default_settings

* Improvements

* Remove unused logger

* Run pre-commit

* Restore and improve duplicate key handling and warning

* type → object

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-07 00:17:55 +04:00
Albert Eduardovich N. 294ee051cc
Reducing memory footprint of `Request` object: __slots__ and lazy evaluation (#7036)
* reducing memory foorprint of `Request` object

* restore flow

* Test setters

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:15:53 +04:00
Andrey Rakhmatullin 2e53d90e4c
Refactor HTTP download handlers (#7238) 2026-02-03 20:13:30 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin 54d8562fb0
Assorted test improvements (#7232)
* Await some missed deferreds.

* Simplify test_engine_stop_download_*.

* Fix the header value.

* Extend TestHttpWithCrawlerBase.

* Make test_tls_logging more universal.

* Add more tests for header handling.

* Clarify certificate and ip_address integration tests.

* Make test_download_with_maxsize_very_large_file() more universal.

* Fix test_coroutine_asyncio().

* Typo.
2026-02-02 09:37:39 +01:00
Andrey Rakhmatullin 4e1faf883d
Foundations for the reactorless mode. (#7199)
* Foundations for the reactorless mode.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
2026-01-31 20:48:54 +05:00
Andrey Rakhmatullin 3ec6ae05c1
Fix omitting repeated dataloss warnings in HTTP11DownloadHandler. (#7222) 2026-01-22 00:41:27 +05:00
Andrey Rakhmatullin 2347138ba4
Unified download handler exceptions (#7208)
* Add DownloadTimeoutError.

* Don't expect ConnectionAborted in tests.

* Add DownloadCancelledError.

* Add ResponseDataLoss.

* Add more download handler tests.

* Add DownloadConnectionRefusedError.

* Add UnsupportedURLScheme.

* Add CannotResolveHostError.

* Add DownloadFailedError.

* Remove wrapped Twisted exceptions from lists.

* Update a test expectation.

* Extract wrap_twisted_exceptions().

* Wrap TxTimeoutError.

* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
Adrian Chaves ba3d7bc7a8 Remove the non-standard 307/308 handling, and align other aspects with the standard 2026-01-21 18:46:05 +01:00
Andrey Rakhmatullin 9bae1ee21f
Lazy creation of Downloader._slot_gc_loop. (#7210) 2026-01-20 12:58:09 +01:00
Adrián Chaves 842d0becf0
Rename test function 2026-01-20 12:15:25 +01:00
Adrian Chaves f7f18123eb Remove body on 307/308 redirects and switch to GET on 301 2026-01-19 19:15:20 +01:00
Adrian Chaves b6e5c58ae7 Fix arbitrary callable execution from the Referrer-Policy response header 2026-01-19 16:44:03 +01:00
Andrey Rakhmatullin 0f07b2e38c
Pass the spider arg to custom stat collectors {open,close}_spider(). (#7213) 2026-01-09 16:57:53 +01:00