RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,227 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

2175 Commits

Author SHA1 Message Date
mayank-dev-15 bed79d0034 fix: stabilize test_start_requests_laziness race condition 
Remove racy ordering assertion that depends on exact download timing.
The spider's own assert in start_requests already verifies laziness -
it fails if all 100 start requests are consumed before any download.
The test now verifies both initial seeds and follow-up requests completed.
 2026-07-04 16:33:12 +05:30
Adrián Chaves 36287cb665 Merge branch 'redirect-protocols' into 2.11 2024-05-13 20:09:00 +02:00
Adrián Chaves f138d5d145 Merge branch 'environ-proxy-protocol' into 2.11 2024-05-13 20:06:15 +02:00
Adrián Chaves 1d0502f25b Merge branch 'advisory-fix' into 2.11 2024-05-13 19:51:49 +02:00
Adrián Chaves bb948af00b
Release notes for 2.11.2 (#6359) 2024-05-13 19:49:37 +02:00
Adrián Chaves 5ad9433dd5 Merge remote-tracking branch 'scrapy/2.11' into 2.11 2024-05-13 16:27:40 +02:00
Adrián Chaves a6c89a2e0a scrapy parse: fix the signature of callbacks from the CLI (#6182) 2024-05-13 16:25:07 +02:00
Adrián Chaves 67c77eb9f2 MediaPipeline: log media_to_download errors before stripping them (#5068) 2024-05-13 16:24:51 +02:00
Bagaudin Magomedov fa4c42dfd7 Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 16:22:46 +02:00
Lorenzo Verardo 1f4523f1a2 Handle robots.txt files not UTF-8 encoded (#6298) 2024-05-13 16:21:57 +02:00
Víctor Ruiz 915f40fa81 Fix WrappedRequest.get_header raising TypeError if default is None (#6310) 2024-05-13 16:21:00 +02:00
Jalil SA 8c0bc1b698 fix: LxmlLinkExtractor unique_list missing key (#6221) 2024-05-13 16:19:52 +02:00
Adrián Chaves 82981fb8a2 Do not close the underlying file from compression plugins (#6239) 2024-05-13 16:10:19 +02:00
Adrián Chaves f149ea4b80
Fix the offsite middleware missing some requests (#6358) 2024-05-13 15:32:53 +02:00
aisha-partha 3562618f67 Set METAREFRESH_IGNORE_TAGS to ['noscript'] by default (#6347) 2024-05-13 14:23:15 +02:00
Andrey Rakhmatullin 397d21f1f5 Cherry-pick WindowsRunSpiderCommandTest fixes (#6290). 2024-05-08 16:42:30 +05:00
kokobhara bb4ff4d5cd Fix test expectations (#6316) 2024-04-27 20:23:35 +05:00
Adrián Chaves 7a1ab7e1be Do not drop Cookie on http → http 2024-03-06 16:19:19 +01:00
Adrián Chaves 6499214a4f Keep the Cookie header on scheme upgrades 2024-03-06 15:40:13 +01:00
Tsubasa Umeuchi f8d6c456e0 Fixed logic for handling headers on redirect 2024-03-02 14:02:27 +09:00
Adrián Chaves c5446c0b85 Fix proxy handling on redirect to different scheme 2024-02-27 22:41:27 +01:00
Adrián Chaves 685cf5940f Fix MetaRefreshMiddleware 2024-02-20 13:57:06 +01:00
Adrián Chaves 33ef5450f9 Fix RedirectMiddleware 2024-02-20 13:36:59 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Adrián Chaves c5dad41190 Speed up tests, remove comments without regexps 2024-02-02 14:03:16 +01:00
Andrey Rakhmatullin 09a7efef7c Remove a defer.returnValue call. 2024-01-15 18:19:29 +04:00
Adrián Chaves 27781a85e7 Fix bad closing tags in XMLFeedSpider tests 2023-12-15 13:52:12 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 9655b0b8eb Mark slow tests, with their own tox env and CI job 2023-12-15 10:19:47 +01:00
Adrián Chaves 4f72b49f97 Fix namespaces nodename support for xmliter_lxml 2023-12-15 10:06:13 +01:00
Laerte Pereira b9c4ee26d7 Remove deprecated scrapy.downloadermiddlewares.decompression 2023-12-13 13:28:30 +01:00
Adrián Chaves 1533b69032 Test and address ReDoS attack vectors for open_in_browser 2023-12-13 12:01:35 +01:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 3fda2fe103 Protect against gzip bomb sitemaps 2023-11-22 18:34:37 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Adrián Chaves 1045856a50 Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-11-16 19:16:52 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 538192916f Merge pull request #6064 from wRAR/signals-proper 
Refactor installing signals.
 2023-11-16 19:15:30 +04:00
Andrey Rakhmatullin 59cfdeaa5c Merge pull request #6083 from wRAR/py3.12-release 
Adapt to the Python 3.12 final release
 2023-11-16 19:14:47 +04:00
Adrián Chaves 080fecd890 Drop the Authorization header on cross-domain redirect 2023-11-15 15:39:30 +01:00