Commit Graph

11070 Commits

Author SHA1 Message Date
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468)
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Adrian 3561748280 sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 13:20:30 +05:00
Andrey Rakhmatullin b7cd42da39 Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:02:01 +05:00
Adrian da6dfae750 Restore 2.14 getwithbase, add a new method for class key deduplication (#7449)
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
2026-04-22 19:01:59 +05:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373)
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
2026-04-09 16:56:30 +05:00
Adrian b68f26726a
Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
Andrey Rakhmatullin 2b174e348d
Silence the CertificateOptions method warning. (#7410)
* Silence the CertificateOptions method warning.

* Fix typing.
2026-04-08 18:54:13 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395)
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Albert Eduardovich N. 9fffcc1b82
Optimize SitemapSpider memory usage (#7007) 2026-04-08 13:59:16 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Albert Eduardovich N. 8835a69f12
Use `dataclass` for the downloader `Slot` (#7405) 2026-04-08 13:26:01 +02:00
Ahmed Arshad 830eaeab5d
docs: fix typos in addons.rst (#7408)
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
2026-04-08 09:38:09 +02:00
Adrian 010faf1722
Use sphinx-scrapy 0.7.1 (#7406) 2026-04-06 15:29:59 +02:00
Andrey Rakhmatullin 8a26c3c2a0
Use a random port in MockFTPServer. (#7402) 2026-04-06 14:55:33 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386)
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
Albert Eduardovich N. e3a8ff2b59
improve trackref (#7375)
* improve trackref

- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`

* tracing GC is no joke

* refine tests

* explain the tests

* add note for pypy in docs + ...

return empty tuple instead of list in `iter_all`
2026-04-03 15:15:30 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 510f09a961
Deprecate ScrapyClientContextFactory. (#7391) 2026-04-02 09:29:47 +02:00
Albert Eduardovich N. ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` (#7388)
* deprecate `walk_modules` in favour of `walk_modules_iter`

* remove deprecation line from docstring

* addressing review
2026-04-01 22:11:35 +05:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385)
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. (#7387)
* Move TLS logging from _ScrapyClientTLSOptions to handlers.

* Update the docs.

* Fix typing.

* Update new docs.

* Add the comment back.
2026-04-01 16:02:55 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Albert Eduardovich N. 72bcf8cb46
adjust httpx error mapping (#7384) 2026-03-31 19:02:38 +02:00
Adrian Chaves c4c0555ccf Enable parallel execution in the other internal Sphinx plugin 2026-03-30 18:01:29 +02:00
Adrian Chaves 299993b62a Fix the docs build 2026-03-30 17:46:55 +02:00
Adrian Chaves 74c33e5172 Use the spam label when auto-closing PRs 2026-03-30 17:43:59 +02:00
Adrian 6cef717dad
Add a template for PRs (#7381) 2026-03-30 17:33:21 +02:00
Kailesh 86a7ceaa9f
Dynamic loading for S3 HTTPS handler (#7370) 2026-03-30 10:12:43 +02:00
Andrey Rakhmatullin 31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed (#7368)
* Add docs for HttpxDownloadHandler.

* Handle httpx not being installed.

* Revert test_not_configured_without_asyncio().

* Restore the comment.

* Cleanup.
2026-03-27 21:06:55 +05:00
Andrey Rakhmatullin fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. (#7366) 2026-03-27 12:21:25 +01:00
Albert Eduardovich N. 5561aaec1d
fixing oversight from #7036 (#7367)
https://github.com/scrapy/scrapy/pull/7269#issuecomment-3958535816
2026-03-27 11:38:44 +01:00
Andrey Rakhmatullin a8e99aeb2e
Fix botocore-pinned and pypy3-* (#7363) 2026-03-26 08:21:58 +01:00
Andrey Rakhmatullin 2ce02d417a
TLS code modernization (#7353)
* Import some stuff from service_identity directly.

* Extract _log_tls().

* Sync validating IP addresses.

* Introduce _deps_compat.

* Deprecate getCertificateOptions().

* Deprecate ScrapyClientTLSOptions.

* Add pragma: no cover.

* verify=False and trustRoot=platformTrust() are the defaults.

* Improve docstrings for TLS classes.

* Deprecate AcceptableProtocolsContextFactory.

* Switch from DefaultOpenSSLContextFactory to CertificateOptions.

* Deprecate load_context_factory_from_settings().

* Remove the outdated warning about incompatible factories.

* Simplify _load_context_factory_from_settings().

* Make CertificateOptions and Context once per factory.

* pragma: no cover
2026-03-25 15:42:43 +05:00
Andrey Rakhmatullin 03d105ac92
Add warnings to components not working without a reactor (#7355) 2026-03-25 09:21:08 +01:00
Kaileshwar R K 54a4c3af89
Deprecate the mail API (#7263) 2026-03-24 21:31:12 +01:00
Andrey Rakhmatullin bfe34492fa
asyncio alternative to deferToThread() (#7349) 2026-03-24 15:41:15 +01:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin d42b23d78a
Await persist_file() in media pipelines. (#7182) 2026-03-24 10:52:56 +01:00
Andrey Rakhmatullin 9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. (#7276) 2026-03-24 10:36:48 +01:00
Andrey Rakhmatullin 939db88b04
Disable mypy allow_untyped_calls. (#7331) 2026-03-24 10:31:17 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Andrey Rakhmatullin 584d99af30
Split long test files (#7329) 2026-03-13 10:30:25 +01:00