Commit Graph

2548 Commits

Author SHA1 Message Date
Adrian Chaves b90842379d Extend download error coverage 2026-04-22 17:04:32 +02:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441)
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
2026-04-21 18:27:16 +05:00
Hamzah Alshawwaf 9da14cdff1
Fix file extension extraction for URLs with query parameters (#4225) (#7414)
* Fix file extension extraction for URLs with query parameters (#4225)

* Prioritize parsed path for file extensions and add tests (#4225)
2026-04-14 20:04:38 +05:00
Adrian b68f26726a
Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395)
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Albert Eduardovich N. 9fffcc1b82
Optimize SitemapSpider memory usage (#7007) 2026-04-08 13:59:16 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Albert Eduardovich N. 8835a69f12
Use `dataclass` for the downloader `Slot` (#7405) 2026-04-08 13:26:01 +02:00
Andrey Rakhmatullin 8a26c3c2a0
Use a random port in MockFTPServer. (#7402) 2026-04-06 14:55:33 +02:00
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386)
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
Albert Eduardovich N. e3a8ff2b59
improve trackref (#7375)
* improve trackref

- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`

* tracing GC is no joke

* refine tests

* explain the tests

* add note for pypy in docs + ...

return empty tuple instead of list in `iter_all`
2026-04-03 15:15:30 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 510f09a961
Deprecate ScrapyClientContextFactory. (#7391) 2026-04-02 09:29:47 +02:00
Albert Eduardovich N. ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` (#7388)
* deprecate `walk_modules` in favour of `walk_modules_iter`

* remove deprecation line from docstring

* addressing review
2026-04-01 22:11:35 +05:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385)
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Kailesh 86a7ceaa9f
Dynamic loading for S3 HTTPS handler (#7370) 2026-03-30 10:12:43 +02:00
Andrey Rakhmatullin fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. (#7366) 2026-03-27 12:21:25 +01:00
Albert Eduardovich N. 5561aaec1d
fixing oversight from #7036 (#7367)
https://github.com/scrapy/scrapy/pull/7269#issuecomment-3958535816
2026-03-27 11:38:44 +01:00
Andrey Rakhmatullin 2ce02d417a
TLS code modernization (#7353)
* Import some stuff from service_identity directly.

* Extract _log_tls().

* Sync validating IP addresses.

* Introduce _deps_compat.

* Deprecate getCertificateOptions().

* Deprecate ScrapyClientTLSOptions.

* Add pragma: no cover.

* verify=False and trustRoot=platformTrust() are the defaults.

* Improve docstrings for TLS classes.

* Deprecate AcceptableProtocolsContextFactory.

* Switch from DefaultOpenSSLContextFactory to CertificateOptions.

* Deprecate load_context_factory_from_settings().

* Remove the outdated warning about incompatible factories.

* Simplify _load_context_factory_from_settings().

* Make CertificateOptions and Context once per factory.

* pragma: no cover
2026-03-25 15:42:43 +05:00
Andrey Rakhmatullin 03d105ac92
Add warnings to components not working without a reactor (#7355) 2026-03-25 09:21:08 +01:00
Kaileshwar R K 54a4c3af89
Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
Andrey Rakhmatullin bfe34492fa
asyncio alternative to deferToThread() (#7349) 2026-03-24 15:41:15 +01:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Andrey Rakhmatullin 9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. (#7276) 2026-03-24 10:36:48 +01:00
Andrey Rakhmatullin 939db88b04
Disable mypy allow_untyped_calls. (#7331) 2026-03-24 10:31:17 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Andrey Rakhmatullin 584d99af30
Split long test files (#7329) 2026-03-13 10:30:25 +01:00
Adrian Chaves 9dfe449d13 Fix test issues with the no-reactor env 2026-03-12 16:46:32 +01:00
Adrian Chaves 4e3df249f2 Merge branch '2.14' 2026-03-12 16:02:37 +01:00
Adrian Chaves 8e28f938d2 Make test_no_warning_when_referer_middleware_present less brittle 2026-03-12 14:29:51 +01:00
Adrian Chaves 886131c7b2 Run pre-commit 2026-03-12 13:54:00 +01:00
Adrian Chaves 945b787a26 Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14 2026-03-12 13:37:24 +01:00
Andrey Rakhmatullin e02ad08672
Move to mypy --strict with exceptions. (#7300)
* Move to mypy --strict with exceptions.

* Fix typing-tests.
2026-03-02 15:47:23 +05:00
Andrey Rakhmatullin 7010985e4f
Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin abd025f78e
Pass correct env to PopenSpawn() in tests. (#7279) 2026-02-23 11:34:56 +01:00
pierreeurope 3fe89a211b
Fix open_spider exception handling to set exit code 1 (#7255) 2026-02-19 11:21:59 +01:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 6e0a0e476a
Fix running tests with -n auto, enable it on CI (#7257)
* Make test_start_deprecated_super() more robust.

* Work around the FTPFilesStore.FTP_* initialization.

* Enable xdist on CI.

* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
Andrey Rakhmatullin 09bd8f4231
Properly close ftplib.FTP(). (#7256) 2026-02-13 19:12:13 +05:00
Andrey Rakhmatullin 0e1526ed30
Fix flaky test_download_with_proxy_https_timeout(). (#7250) 2026-02-13 09:45:09 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Ryotaro 06fb87f7bb
Fix override behavior in getwithbase() issue #6912 (#6993)
* Fix test for getwithbase() to ensure class-keyed overrides with None are handled correctly

* Test actual import paths from default_settings

* Improvements

* Remove unused logger

* Run pre-commit

* Restore and improve duplicate key handling and warning

* type → object

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-07 00:17:55 +04:00
Albert Eduardovich N. 294ee051cc
Reducing memory footprint of `Request` object: __slots__ and lazy evaluation (#7036)
* reducing memory foorprint of `Request` object

* restore flow

* Test setters

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
2026-02-05 22:15:53 +04:00
Andrey Rakhmatullin 2e53d90e4c
Refactor HTTP download handlers (#7238) 2026-02-03 20:13:30 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00