RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,102 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

11102 Commits

Author SHA1 Message Date
Adrian Chaves 1e6aca0bff 🤞 2026-04-29 22:22:29 +02:00
Adrian Chaves c5250f1d80 Remove sleep that may have been making a test brittle 2026-04-29 21:43:48 +02:00
Adrian Chaves 1b917950f5 Remove unneeded comment 2026-04-29 16:40:41 +02:00
Adrian Chaves f5080c1e4c Minor refactoring 2026-04-29 16:38:32 +02:00
Adrian Chaves 73f7dbaa50 Merge remote-tracking branch 'origin/master' into faster-shutdown-2 2026-04-29 16:31:53 +02:00
Adrian Chaves dc4db5318f Address typing issue 2026-04-29 16:23:23 +02:00
Adrian Chaves de950c9fa1 Improve coverage 2026-04-29 16:06:59 +02:00
Adrian Chaves 2a0825d75a Address typing and linting issues 2026-04-29 15:18:05 +02:00
Adrian Chaves dbcd87c09c Improve test coverage 2026-04-29 14:52:17 +02:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Adrian Chaves fe8f527b1a Fix test for the default reactor 2026-04-29 13:39:39 +02:00
Adrian Chaves 17c406d91b Improve coverage 2026-04-29 13:12:41 +02:00
Adrian Chaves 41fa1e94a0 Add CI timeouts while at it 2026-04-29 12:45:52 +02:00
Adrian Chaves 03f89c99ab Remove unused code and privatize to protect against import of imports from public modules 2026-04-29 12:45:16 +02:00
Adrian Chaves 44f0fd0d37 Solve issue making CI hang? 2026-04-29 12:25:54 +02:00
Adrian Chaves 2b397afe52 Merge remote-tracking branch 'origin/master' into faster-shutdown-2 2026-04-29 11:52:12 +02:00
Albert Eduardovich N. dd36fb7859
Improve handling of time deltas (#7377) 2026-04-29 09:53:40 +02:00
Andrey Rakhmatullin 30a54b72f0
Merge pull request #7470 from scrapy/2.15 
Merge 2.15.2 into the main branch
 2026-04-28 21:19:29 +05:00
Adrian Chaves 3d5ca9f433 Bump version: 2.15.1 → 2.15.2 2026-04-28 15:28:51 +02:00
Adrian 3a88cd0e2b
2.15.2 (#7468) 
* sphinx-scrapy: 0.8.3 → 0.8.4 (#7467)

* Release notes for 2.15.2
 2026-04-28 15:26:42 +02:00
Andrey Rakhmatullin 988afe1454
Clarify dataloss/http2 tests. (#7466) 2026-04-28 14:35:41 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Andrey Rakhmatullin 068aa69b35
Fixes for Python 3.14. (#7460) 2026-04-27 18:47:52 +05:00
Adrian Chaves 32a69f9547 Merge remote-tracking branch 'origin/master' into faster-shutdown-2 2026-04-27 15:30:56 +02:00
Adrian Chaves 15af004382 Address typing and linting issues 2026-04-27 15:28:27 +02:00
Andrey Rakhmatullin fc4c57e795
Fix the substitution pattern in open_in_browser(). (#7459) 
* Fix the substitution pattern in open_in_browser().

* Formatting.
 2026-04-24 13:54:37 +05:00
Andrey Rakhmatullin 4e25686b20 Merge branch '2.15' 2026-04-23 21:08:38 +05:00
Andrey Rakhmatullin f3c5a6e75f Bump version: 2.15.0 → 2.15.1 2026-04-23 19:57:44 +05:00
Andrey Rakhmatullin 416a454dc5
Release notes for 2.15.1. (#7456) 2026-04-23 19:51:00 +05:00
Adrian Chaves 8613d390d7 Support fast crawler stops 2026-04-23 13:46:10 +02:00
Adrian 3561748280 sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 13:20:30 +05:00
Adrian 41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 10:03:09 +02:00
Andrey Rakhmatullin b7cd42da39 Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:02:01 +05:00
Adrian da6dfae750 Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
 2026-04-22 19:01:59 +05:00
Andrey Rakhmatullin 320e40a044
Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:01:01 +05:00
Adrian 294abed138
Restore 2.14 getwithbase, add a new method for class key deduplication (#7449) 
* Restore 2.14 getwithbase, add a new method for class key deduplication

* Use the original getwithbase code, not some equivalent

* Fix mocking

* Test key exceptions
 2026-04-22 19:00:39 +05:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441) 
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
 2026-04-21 18:27:16 +05:00
Hamzah Alshawwaf 9da14cdff1
Fix file extension extraction for URLs with query parameters (#4225) (#7414) 
* Fix file extension extraction for URLs with query parameters (#4225)

* Prioritize parsed path for file extensions and add tests (#4225)
 2026-04-14 20:04:38 +05:00
NI283105 508367664f
Fix missing load_object import and grammar in docs (#7417) 
Co-authored-by: NI <Nidotexe@gmail.com>
 2026-04-09 15:31:01 +02:00
Andrey Rakhmatullin 47e25fbbb5 Bump version: 2.14.2 → 2.15.0 2026-04-09 17:00:40 +05:00
Andrey Rakhmatullin 1432455d35 Add a missing release notes change. 2026-04-09 16:59:18 +05:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Adrian b68f26726a
Fix DownloaderAwarePriorityQueue tie-breaking across slots (#7351) 2026-04-08 16:10:21 +02:00
Andrey Rakhmatullin 2b174e348d
Silence the CertificateOptions method warning. (#7410) 
* Silence the CertificateOptions method warning.

* Fix typing.
 2026-04-08 18:54:13 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Albert Eduardovich N. 5b37613618
Improve commands (#7376) 2026-04-08 14:59:00 +02:00
Albert Eduardovich N. 13a014d2e6
Response now uses less memory (#7374) 2026-04-08 14:25:11 +02:00
Albert Eduardovich N. 9fffcc1b82
Optimize SitemapSpider memory usage (#7007) 2026-04-08 13:59:16 +02:00
Kailesh a4377f9a4f
Rename DNS_RESOLVER to TWISTED_DNS_RESOLVER (#7361) 2026-04-08 13:58:35 +02:00
Albert Eduardovich N. 8835a69f12
Use `dataclass` for the downloader `Slot` (#7405) 2026-04-08 13:26:01 +02:00