RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 1fab844f7d Pin the Python version for typing-tests. 2023-12-20 15:57:51 +04:00
Adrián Chaves 1864f48e9e Link to Zyte’s export guides 2023-12-20 12:47:34 +01:00
Swati Anshu c67f730695
create_instance → build_from_crawler, build_from_settings (#6169) 2023-12-18 11:51:02 +01:00
Adrián Chaves 27781a85e7 Fix bad closing tags in XMLFeedSpider tests 2023-12-15 13:52:12 +01:00
Adrián Chaves c7c7a488b9 Fix typing issues 2023-12-15 13:18:23 +01:00
Adrián Chaves bc138ef8e9 Minor release notes fix 2023-12-15 12:24:04 +01:00
Adrián Chaves ce9d290eff Remove the lxml version check for huge_tree on xmliter_lxml 
iterparse supports the option since lxml 2.2.1, it was the HTML parser that only got it in 4.2
 2023-12-15 12:23:26 +01:00
Adrián Chaves a49c8762dd Avoid calling iterparse twice 2023-12-15 12:14:53 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves b0dbd0e9af Minor naming changes 2023-12-15 11:49:22 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 2538c0e862 Restore the implementation of xmliter 2023-12-15 10:23:24 +01:00
Adrián Chaves 9655b0b8eb Mark slow tests, with their own tox env and CI job 2023-12-15 10:19:47 +01:00
Adrián Chaves d50f436a73 Enable huge_tree for xmliter_lxml 2023-12-15 10:08:45 +01:00
Adrián Chaves 4f72b49f97 Fix namespaces nodename support for xmliter_lxml 2023-12-15 10:06:13 +01:00
Adrián Chaves 12b10a7a64 Cover scrapy.downloadermiddlewares.decompression in the release notes 2023-12-13 13:35:05 +01:00
Laerte Pereira b9c4ee26d7 Remove deprecated scrapy.downloadermiddlewares.decompression 2023-12-13 13:28:30 +01:00
Adrián Chaves 1533b69032 Test and address ReDoS attack vectors for open_in_browser 2023-12-13 12:01:35 +01:00
Adrián Chaves bb74badd1b spider → mw 2023-12-11 17:39:55 +01:00
Adrián Chaves c66b517706 Add Python 3.13 alpha to CI 2023-12-01 10:36:27 +01:00
Adrián Chaves 70ba3a0868
SPM → Zyte API (#6163) 2023-11-30 11:01:22 +01:00
Adrián Chaves 731f749556
Extend Request.meta documentation (#5565) 2023-11-30 10:54:09 +01:00
Adrián Chaves 40b3efbbee Remove open_in_browser from the 1.8.4 release notes 2023-11-29 12:47:04 +01:00
Adrián Chaves eb8b2c5197 Mention open_in_browser in the release notes 2023-11-29 12:13:04 +01:00
Adrián Chaves c947f51077 Set an arbitrary upper limit on ReDoS-vulnerable regexps 2023-11-29 11:54:08 +01:00
Adrián Chaves a113208a06 Fix BytesIO non-emptiness check 2023-11-24 11:35:15 +01:00
Adrián Chaves 8a73c6c90c Fix HttpCompressionMiddleware backward compatibility 2023-11-24 10:25:01 +01:00
Adrián Chaves 62398e424c CHUNK_SIZE: 8 KiB → 32 KiB 2023-11-24 10:20:14 +01:00
Adrián Chaves 6c278e1862 List[bytes] → BytesIO 2023-11-24 10:15:38 +01:00
Adrián Chaves 5f2827efe7 Make HttpCompressionMiddleware changes backward-comaptible 2023-11-24 10:15:38 +01:00
Andrey Rakhmatullin fa690fbe03
Merge pull request #6154 from Laerte/master 
Remove test_download_gzip_response test
 2023-11-23 20:56:50 +04:00
Laerte Pereira 09ce0ef526 Remove test_download_gzip_response test 2023-11-23 10:50:46 -03:00
Adrián Chaves 8e25f8c157
Fix bad message 2023-11-23 14:12:59 +01:00
Adrián Chaves cf80e5670e Solve linting and typing issues 2023-11-23 12:07:15 +01:00
Adrián Chaves b53ed52a22 Update the release notes 2023-11-23 11:36:45 +01:00
Adrián Chaves 03d9866518 Also use DOWNLOAD_WARNSIZE for decompressions 2023-11-23 10:26:47 +01:00
Adrián Chaves 1087bb7b2e Update the docs 2023-11-23 09:11:14 +01:00
Adrián Chaves e0b66c021a Mind Spider.download_maxsize and Request.meta['download_maxsize'] 2023-11-22 19:03:24 +01:00
Adrián Chaves 3fda2fe103 Protect against gzip bomb sitemaps 2023-11-22 18:34:37 +01:00
Adrián Chaves 9cc8703877 Protect against zstandard bombs 2023-11-22 17:53:00 +01:00
Adrián Chaves fba167c5e1 Protect against brotli bombs 2023-11-22 17:32:09 +01:00
Adrián Chaves 0c4a98f8e0 Protect against deflate bombs 2023-11-22 17:12:43 +01:00
Adrián Chaves 0bf29a7b1b Update test expectations 2023-11-22 16:10:50 +01:00
Adrián Chaves 6969041c5f Protect against gzip bombs 2023-11-22 15:52:00 +01:00
Adrián Chaves 1c4e93293a Merge remote-tracking branch 'scrapy/2.11' into 2.11-authorization 2023-11-17 15:42:08 +01:00
Kiran 49b284ab85 Updated README.rst (#6144) 2023-11-16 19:17:52 +04:00
Jessica Allman-LaPorte 150f9d6d88 Make shell switching more clear in the tutorial (#6128) 2023-11-16 19:17:29 +04:00
Adrián Chaves 1045856a50 Merge pull request #6112 from wRAR/test-shutdown-forced 
Make shutdown tests more robust.
 2023-11-16 19:16:52 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 538192916f Merge pull request #6064 from wRAR/signals-proper 
Refactor installing signals.
 2023-11-16 19:15:30 +04:00