2b174e348d
Silence the CertificateOptions method warning. ( #7410 )
...
* Silence the CertificateOptions method warning.
* Fix typing.
2026-04-08 18:54:13 +05:00
b9be5ce053
Reactorless shell and other small shell improvements ( #7395 )
...
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
2026-04-08 18:24:51 +05:00
5b37613618
Improve commands ( #7376 )
2026-04-08 14:59:00 +02:00
13a014d2e6
Response now uses less memory ( #7374 )
2026-04-08 14:25:11 +02:00
9fffcc1b82
Optimize SitemapSpider memory usage ( #7007 )
2026-04-08 13:59:16 +02:00
a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER ( #7361 )
2026-04-08 13:58:35 +02:00
8835a69f12
Use `dataclass` for the downloader `Slot` ( #7405 )
2026-04-08 13:26:01 +02:00
830eaeab5d
docs: fix typos in addons.rst ( #7408 )
...
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
2026-04-08 09:38:09 +02:00
010faf1722
Use sphinx-scrapy 0.7.1 ( #7406 )
2026-04-06 15:29:59 +02:00
8a26c3c2a0
Use a random port in MockFTPServer. ( #7402 )
2026-04-06 14:55:33 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
58d85282cf
chore: more ruff rules and overall minor improvements ( #7386 )
...
* overall prettyfication
- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting
* return `SIM300` to ignore
* apply ruff rules to all files
* another one
* remove extra space in pyproject.toml
* simplify `__getattr__` in `scrapy.utils.url`
* lazy `url_is_from_any_domain` and `url_is_from_spider`
* improve typing
* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
e3a8ff2b59
improve trackref ( #7375 )
...
* improve trackref
- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`
* tracing GC is no joke
* refine tests
* explain the tests
* add note for pypy in docs + ...
return empty tuple instead of list in `iter_all`
2026-04-03 15:15:30 +05:00
b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. ( #7394 )
2026-04-02 10:20:54 +02:00
510f09a961
Deprecate ScrapyClientContextFactory. ( #7391 )
2026-04-02 09:29:47 +02:00
ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` ( #7388 )
...
* deprecate `walk_modules` in favour of `walk_modules_iter`
* remove deprecation line from docstring
* addressing review
2026-04-01 22:11:35 +05:00
0c6ccf50b3
Docs for the reactorless mode ( #7385 )
...
* Reactorless mode docs.
* Unset the ftp handler in the reactorless mode.
* Cleanup reactorless subprocess tests.
* Better handling of incompatible TWISTED_ENABLED cases.
* Improvements.
* Improve early error logging in test scripts.
* Address feedback.
* Address feedback.
2026-04-01 20:40:34 +05:00
fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. ( #7387 )
...
* Move TLS logging from _ScrapyClientTLSOptions to handlers.
* Update the docs.
* Fix typing.
* Update new docs.
* Add the comment back.
2026-04-01 16:02:55 +05:00
eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES ( #7379 )
...
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.
* Deprecate BrowserLikeContextFactory.
* Add DOWNLOAD_VERIFY_CERTIFICATES.
* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
72bcf8cb46
adjust httpx error mapping ( #7384 )
2026-03-31 19:02:38 +02:00
c4c0555ccf
Enable parallel execution in the other internal Sphinx plugin
2026-03-30 18:01:29 +02:00
299993b62a
Fix the docs build
2026-03-30 17:46:55 +02:00
74c33e5172
Use the spam label when auto-closing PRs
2026-03-30 17:43:59 +02:00
6cef717dad
Add a template for PRs ( #7381 )
2026-03-30 17:33:21 +02:00
86a7ceaa9f
Dynamic loading for S3 HTTPS handler ( #7370 )
2026-03-30 10:12:43 +02:00
31bf7c3892
Docs for HttpxDownloadHandler, handle httpx not being installed ( #7368 )
...
* Add docs for HttpxDownloadHandler.
* Handle httpx not being installed.
* Revert test_not_configured_without_asyncio().
* Restore the comment.
* Cleanup.
2026-03-27 21:06:55 +05:00
fee20b7858
AsyncCrawlerProcess._start_asyncio() improvements. ( #7366 )
2026-03-27 12:21:25 +01:00
5561aaec1d
fixing oversight from #7036 ( #7367 )
...
https://github.com/scrapy/scrapy/pull/7269#issuecomment-3958535816
2026-03-27 11:38:44 +01:00
a8e99aeb2e
Fix botocore-pinned and pypy3-* ( #7363 )
2026-03-26 08:21:58 +01:00
2ce02d417a
TLS code modernization ( #7353 )
...
* Import some stuff from service_identity directly.
* Extract _log_tls().
* Sync validating IP addresses.
* Introduce _deps_compat.
* Deprecate getCertificateOptions().
* Deprecate ScrapyClientTLSOptions.
* Add pragma: no cover.
* verify=False and trustRoot=platformTrust() are the defaults.
* Improve docstrings for TLS classes.
* Deprecate AcceptableProtocolsContextFactory.
* Switch from DefaultOpenSSLContextFactory to CertificateOptions.
* Deprecate load_context_factory_from_settings().
* Remove the outdated warning about incompatible factories.
* Simplify _load_context_factory_from_settings().
* Make CertificateOptions and Context once per factory.
* pragma: no cover
2026-03-25 15:42:43 +05:00
03d105ac92
Add warnings to components not working without a reactor ( #7355 )
2026-03-25 09:21:08 +01:00
54a4c3af89
Deprecate the mail API ( #7263 )
2026-03-24 21:31:12 +01:00
bfe34492fa
asyncio alternative to deferToThread() ( #7349 )
2026-03-24 15:41:15 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
d42b23d78a
Await persist_file() in media pipelines. ( #7182 )
2026-03-24 10:52:56 +01:00
9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. ( #7276 )
2026-03-24 10:36:48 +01:00
939db88b04
Disable mypy allow_untyped_calls. ( #7331 )
2026-03-24 10:31:17 +01:00
c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers ( #7283 )
2026-03-19 09:16:40 +01:00
584d99af30
Split long test files ( #7329 )
2026-03-13 10:30:25 +01:00
4d2071f7b3
Use fancy kwarg typing
2026-03-12 16:49:29 +01:00
9dfe449d13
Fix test issues with the no-reactor env
2026-03-12 16:46:32 +01:00
4e3df249f2
Merge branch '2.14'
2026-03-12 16:02:37 +01:00
498b4fc1a4
Bump version: 2.14.1 → 2.14.2
2026-03-12 15:45:16 +01:00
378bb68039
Proofread the release notes
2026-03-12 15:44:54 +01:00
8e28f938d2
Make test_no_warning_when_referer_middleware_present less brittle
2026-03-12 14:29:51 +01:00
886131c7b2
Run pre-commit
2026-03-12 13:54:00 +01:00
945b787a26
Merge remote-tracking branch 'cwxj-rr6w-m6w7/fix-referer-policy-handling' into 2.14
2026-03-12 13:37:24 +01:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
7010985e4f
Bump ruff, fix some of rules ( #7277 )
2026-02-23 11:48:38 +01:00
abd025f78e
Pass correct env to PopenSpawn() in tests. ( #7279 )
2026-02-23 11:34:56 +01:00
Andrey Rakhmatullin
Albert Eduardovich N.
Kailesh
Ahmed Arshad
Adrian
Julian Hamze