RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,876 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

332 Commits

Author SHA1 Message Date
Adrián Chaves a04e85e08b Re-implement front-load support 2025-03-31 11:49:28 +02:00
Adrián Chaves 9417de1df9 Review uses of seed 2025-03-31 09:53:52 +02:00
Adrián Chaves 85e3dc13f1 Merge branch 'minimal-async-seeds' into async-seeds 2025-03-28 15:54:23 +01:00
Adrián Chaves 5e18a132da Clean up from self-review 2025-03-27 14:03:39 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves 97a0fed020 Document the execution order change of non-generator start_requests, for the sake of completion 2025-03-25 17:27:58 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves cc49ea9f18 Inclued () in …start_requests mentions on news.rst 2025-03-24 10:58:19 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 955b40e415 Increase test reliability and min lxml 2025-03-17 10:13:27 +01:00
Adrián Chaves 28fa70a969 WIP documentation improvements 2025-03-17 01:12:26 +01:00
Adrián Chaves 734f4e9a1a Cover the new error handling in news.rst and cover it with a test 2025-03-15 02:26:55 +01:00
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 223560dca7 Update the changelog 2025-03-13 11:33:45 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Rotzbua ee4f527f47
fix(docs): pillow domain is shut down permanently 
See https://github.com/python-pillow/Pillow/issues/8585
 2025-01-19 14:58:02 +01:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin efb53aafdc Fix a typo that broke PDF builds. 2024-11-18 15:44:52 +05:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226) 
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-11-18 13:07:32 +05:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Andrey Rakhmatullin 677e977207 Remove dead links to the Reppy doc from the release notes. 2024-10-18 00:03:32 +05:00
mmoriniere 46cddc6ecf
Ignore SyntaxError as well when SPIDER_LOADER_WARN_ONLY is set to True (#6484) 2024-10-02 10:04:03 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 6b88b3346c Set the release date of versions 2.11.1 and 1.8.4 2024-02-14 18:16:40 +01:00
Adrián Chaves 479619b340 Merge branch '2.11-redos' into 2.11 2024-02-14 17:18:26 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Adrián Chaves 5bcb8fd501 Merge branch '2.11-authorization' into 2.11 2024-02-14 17:14:29 +01:00
Andrey Rakhmatullin a55e933c11
Release notes for 2.11.1 (#6150) 2024-02-14 17:08:40 +01:00
Andy 1c9d308acc
Cover the deprecation and removal of response_httprepr in the release notes (#6216) 2024-02-05 12:52:01 +01:00
Adrián Chaves bc138ef8e9 Minor release notes fix 2023-12-15 12:24:04 +01:00
Adrián Chaves ce9d290eff Remove the lxml version check for huge_tree on xmliter_lxml 
iterparse supports the option since lxml 2.2.1, it was the HTML parser that only got it in 4.2
 2023-12-15 12:23:26 +01:00
Adrián Chaves cd6846b763 Merge remote-tracking branch 'scrapy/2.11' into 2.11-redos 2023-12-15 11:52:35 +01:00
Adrián Chaves 150d96764b Deprecate xmliter in favor of xmliter_lxml 2023-12-15 11:42:55 +01:00
Adrián Chaves 12b10a7a64 Cover scrapy.downloadermiddlewares.decompression in the release notes 2023-12-13 13:35:05 +01:00
Adrián Chaves 40b3efbbee Remove open_in_browser from the 1.8.4 release notes 2023-11-29 12:47:04 +01:00
Adrián Chaves eb8b2c5197 Mention open_in_browser in the release notes 2023-11-29 12:13:04 +01:00
Adrián Chaves c947f51077 Set an arbitrary upper limit on ReDoS-vulnerable regexps 2023-11-29 11:54:08 +01:00