Commit Graph

11085 Commits

Author SHA1 Message Date
Sanjay M a9324fbf76
Fix small documentation wording issues (#7480) 2026-05-04 08:43:21 +02:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Albert Eduardovich N. dd36fb7859
Improve handling of time deltas (#7377) 2026-04-29 09:53:40 +02:00
Andrey Rakhmatullin 30a54b72f0
Merge pull request #7470 from scrapy/2.15
Merge 2.15.2 into the main branch
2026-04-28 21:19:29 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468)
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin 988afe1454
Clarify dataloss/http2 tests. (#7466) 2026-04-28 14:35:41 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Andrey Rakhmatullin 068aa69b35
Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
Andrey Rakhmatullin fc4c57e795
Fix the substitution pattern in open_in_browser(). (#7459)
* Fix the substitution pattern in open_in_browser().

* Formatting.
2026-04-24 13:54:37 +05:00
Andrey Rakhmatullin 4e25686b20 Merge branch '2.15' 2026-04-23 21:08:38 +05:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Adrian 3561748280 sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 13:20:30 +05:00
Adrian 41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 10:03:09 +02:00
Andrey Rakhmatullin b7cd42da39 Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:02:01 +05:00
Adrian da6dfae750 Restore 2.14 getwithbase, add a new method for class key deduplication (#7449)
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
2026-04-22 19:01:59 +05:00
Andrey Rakhmatullin 320e40a044
Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:01:01 +05:00
Adrian 294abed138
Restore 2.14 getwithbase, add a new method for class key deduplication (#7449)
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
2026-04-22 19:00:39 +05:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441)
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
2026-04-21 18:27:16 +05:00
Hamzah Alshawwaf 9da14cdff1
Fix file extension extraction for URLs with query parameters (#4225) (#7414)
* Fix file extension extraction for URLs with query parameters (#4225)

* Prioritize parsed path for file extensions and add tests (#4225)
2026-04-14 20:04:38 +05:00
NI283105 508367664f
Fix missing load_object import and grammar in docs (#7417)
Co-authored-by: NI <Nidotexe@gmail.com>
2026-04-09 15:31:01 +02:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373)
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
2026-04-09 16:56:30 +05:00
Adrian b68f26726a
Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
Andrey Rakhmatullin 2b174e348d
Silence the CertificateOptions method warning. (#7410)
* Silence the CertificateOptions method warning.

* Fix typing.
2026-04-08 18:54:13 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395)
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Albert Eduardovich N. 9fffcc1b82
Optimize SitemapSpider memory usage (#7007) 2026-04-08 13:59:16 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Albert Eduardovich N. 8835a69f12
Use `dataclass` for the downloader `Slot` (#7405) 2026-04-08 13:26:01 +02:00
Ahmed Arshad 830eaeab5d
docs: fix typos in addons.rst (#7408)
Co-authored-by: Abdul Rahman <abdouls@hotmail.com>
2026-04-08 09:38:09 +02:00
Adrian 010faf1722
Use sphinx-scrapy 0.7.1 (#7406) 2026-04-06 15:29:59 +02:00
Andrey Rakhmatullin 8a26c3c2a0
Use a random port in MockFTPServer. (#7402) 2026-04-06 14:55:33 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Albert Eduardovich N. 58d85282cf
chore: more ruff rules and overall minor improvements (#7386)
* overall prettyfication

- don't create empty mutable containers (lists, dicts) where it is appropriate
- removed from ignore section and applied some rules from ruff (but keep them ignored in tests)
- use `deque` in `_AsyncCooperatorAdapter` instead of `list.pop(0)`
- remove `f` prefix from strings without any formatting

* return `SIM300` to ignore

* apply ruff rules to all files

* another one

* remove extra space in pyproject.toml

* simplify `__getattr__` in `scrapy.utils.url`

* lazy `url_is_from_any_domain` and `url_is_from_spider`

* improve typing

* specify `arg_to_iter` with None as arg
2026-04-03 19:00:13 +05:00
Albert Eduardovich N. e3a8ff2b59
improve trackref (#7375)
* improve trackref

- use `NoneType` from `types` since python 3.9 is no longer supported
- use `monotonic` instead of `time` and fix flakiness of `get_oldest`

* tracing GC is no joke

* refine tests

* explain the tests

* add note for pypy in docs + ...

return empty tuple instead of list in `iter_all`
2026-04-03 15:15:30 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 510f09a961
Deprecate ScrapyClientContextFactory. (#7391) 2026-04-02 09:29:47 +02:00
Albert Eduardovich N. ed31dcbb10
deprecate `walk_modules` in favour of `walk_modules_iter` (#7388)
* deprecate `walk_modules` in favour of `walk_modules_iter`

* remove deprecation line from docstring

* addressing review
2026-04-01 22:11:35 +05:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385)
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
2026-04-01 20:40:34 +05:00
Andrey Rakhmatullin fa76ca52e9
Move TLS logging from _ScrapyClientTLSOptions to handlers. (#7387)
* Move TLS logging from _ScrapyClientTLSOptions to handlers.

* Update the docs.

* Fix typing.

* Update new docs.

* Add the comment back.
2026-04-01 16:02:55 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Albert Eduardovich N. 72bcf8cb46
adjust httpx error mapping (#7384) 2026-03-31 19:02:38 +02:00
Adrian Chaves c4c0555ccf Enable parallel execution in the other internal Sphinx plugin 2026-03-30 18:01:29 +02:00
Adrian Chaves 299993b62a Fix the docs build 2026-03-30 17:46:55 +02:00
Adrian Chaves 74c33e5172 Use the spam label when auto-closing PRs 2026-03-30 17:43:59 +02:00
Adrian 6cef717dad
Add a template for PRs (#7381) 2026-03-30 17:33:21 +02:00