RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,206 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

10206 Commits

Author SHA1 Message Date
Adrián Chaves 5ad9433dd5 Merge remote-tracking branch 'scrapy/2.11' into 2.11 2024-05-13 16:27:40 +02:00
Kevin Toms ac5d448054 Add FAQ on making a blank request (#6208) 2024-05-13 16:26:26 +02:00
Adrián Chaves a6c89a2e0a scrapy parse: fix the signature of callbacks from the CLI (#6182) 2024-05-13 16:25:07 +02:00
Adrián Chaves 67c77eb9f2 MediaPipeline: log media_to_download errors before stripping them (#5068) 2024-05-13 16:24:51 +02:00
nihilisticneuralnet 1d6d2a8751 [Solved] JOBDIR= None for when Scheduler initializes disk queue even if JOBDIR is empty string (#6124) 
Co-authored-by: John Doe <johndoe@email.com>
 2024-05-13 16:23:10 +02:00
Bagaudin Magomedov fa4c42dfd7 Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 16:22:46 +02:00
Sanchay Kumar 7ad9eb64c4 Indicate that Selector.type can be json (#6334) 2024-05-13 16:22:31 +02:00
Lorenzo Verardo 1f4523f1a2 Handle robots.txt files not UTF-8 encoded (#6298) 2024-05-13 16:21:57 +02:00
Víctor Ruiz 915f40fa81 Fix WrappedRequest.get_header raising TypeError if default is None (#6310) 2024-05-13 16:21:00 +02:00
Jalil SA 8c0bc1b698 fix: LxmlLinkExtractor unique_list missing key (#6221) 2024-05-13 16:19:52 +02:00
Andrey Rakhmatullin 454bd13a8d Remove the auto-generated copyright years from the docs footer. (#6322) 2024-05-13 16:16:34 +02:00
Adrián Chaves b25f34df69 Bandit: allow-list lxml usages (#6265) 2024-05-13 16:13:34 +02:00
Adrián Chaves 82981fb8a2 Do not close the underlying file from compression plugins (#6239) 2024-05-13 16:10:19 +02:00
Adrián Chaves f149ea4b80
Fix the offsite middleware missing some requests (#6358) 2024-05-13 15:32:53 +02:00
aisha-partha 3562618f67 Set METAREFRESH_IGNORE_TAGS to ['noscript'] by default (#6347) 2024-05-13 14:23:15 +02:00
Andrey Rakhmatullin 397d21f1f5 Cherry-pick WindowsRunSpiderCommandTest fixes (#6290). 2024-05-08 16:42:30 +05:00
Andrey Rakhmatullin 665c84e92c Update MANIFEST.in. 2024-05-08 12:44:50 +05:00
kokobhara bb4ff4d5cd Fix test expectations (#6316) 2024-04-27 20:23:35 +05:00
Adrián Chaves 070f13e7c7
Update practices.rst 2024-04-11 12:18:56 +02:00
Yuri H. Galvao 22ee980dd0 Restore brotlipy support (#6261) 2024-03-01 11:40:10 +01:00
Adrián Chaves 1932722277 Use defusedxml.xmlrpc 2024-02-29 09:50:29 +01:00
Adrián Chaves 11b610dfda
Address tox issues (#6240) 
* Force Python 3.8 in pinned tox envs

* Force werkzeug < 3 when using mitmproxy
 2024-02-21 14:28:39 +01:00
Adrián Chaves 2f1d345e74 Solve test issues 2024-02-14 18:59:01 +01:00
Adrián Chaves 502addc717 Bump version: 2.11.0 → 2.11.1 2024-02-14 18:17:48 +01:00
Adrián Chaves 6b88b3346c Set the release date of versions 2.11.1 and 1.8.4 2024-02-14 18:16:40 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Andrey Rakhmatullin a55e933c11
Release notes for 2.11.1 (#6150) 2024-02-14 17:08:40 +01:00
Adrián Chaves 5e5a92026e Remove slow leftovers 2024-02-02 14:06:45 +01:00
Adrián Chaves 810aaa637d Undo an unintended change 2024-02-02 14:04:28 +01:00
Adrián Chaves c5dad41190 Speed up tests, remove comments without regexps 2024-02-02 14:03:16 +01:00
Andrey Rakhmatullin 09a7efef7c Remove a defer.returnValue call. 2024-01-15 18:19:29 +04:00
Adrián Chaves 6127f7d278 Update quotes.toscrape.com page copies (#6190) 2024-01-01 22:58:34 +04:00
Chan Sau Yee badc7c5be9 Update black reference in docs (#6192) 2024-01-01 22:58:21 +04:00
Adrián Chaves 48a9a58ff2 Link to Zyte’s export guides 2023-12-20 21:43:21 +04:00
Adrián Chaves 369712ee50 SPM → Zyte API (#6163) 2023-12-20 21:43:13 +04:00
Adrián Chaves b095dd218f Extend Request.meta documentation (#5565) 2023-12-20 21:43:03 +04:00
Adrián Chaves 27781a85e7 Fix bad closing tags in XMLFeedSpider tests 2023-12-15 13:52:12 +01:00
Adrián Chaves c7c7a488b9 Fix typing issues 2023-12-15 13:18:23 +01:00
Adrián Chaves bc138ef8e9 Minor release notes fix 2023-12-15 12:24:04 +01:00
Adrián Chaves ce9d290eff Remove the lxml version check for huge_tree on xmliter_lxml 
iterparse supports the option since lxml 2.2.1, it was the HTML parser that only got it in 4.2
 2023-12-15 12:23:26 +01:00
Adrián Chaves a49c8762dd Avoid calling iterparse twice 2023-12-15 12:14:53 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves b0dbd0e9af Minor naming changes 2023-12-15 11:49:22 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 2538c0e862 Restore the implementation of xmliter 2023-12-15 10:23:24 +01:00
Adrián Chaves 9655b0b8eb Mark slow tests, with their own tox env and CI job 2023-12-15 10:19:47 +01:00
Adrián Chaves d50f436a73 Enable huge_tree for xmliter_lxml 2023-12-15 10:08:45 +01:00
Adrián Chaves 4f72b49f97 Fix namespaces nodename support for xmliter_lxml 2023-12-15 10:06:13 +01:00