RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,818 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

10818 Commits

Author SHA1 Message Date
Adrián Chaves 193f170df9 Minor refactoring 2025-03-19 19:31:02 +01:00
Adrián Chaves 1f65d244e5 Finish the page on optimizations 2025-03-17 12:26:19 +01:00
Adrián Chaves 0b9b863bc9 Lower test flakyness 2025-03-17 10:56:47 +01:00
Adrián Chaves 955b40e415 Increase test reliability and min lxml 2025-03-17 10:13:27 +01:00
Adrián Chaves 28fa70a969 WIP documentation improvements 2025-03-17 01:12:26 +01:00
Adrián Chaves 65f3f0d208 WIP documentation improvements 2025-03-16 00:32:23 +01:00
Adrián Chaves ba88283236 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-15 21:20:10 +01:00
Adrián Chaves dac68c685a Fix test_scheduler_next_request_exception 2025-03-15 21:18:13 +01:00
Adrián Chaves b6707ce3bf Address typing issues 2025-03-15 20:58:16 +01:00
Adrián Chaves a5dc229508 Improve exception handling for Scheduler’s has_pending_requests() and enqueue_request() 2025-03-15 20:50:21 +01:00
Andrey Rakhmatullin 8729247213
Fix running simple tests with --reactor=default. (#6735) 2025-03-15 20:33:10 +01:00
Adrián Chaves c8bbe4d968 Log a traceback and assume a None return value if Scheduler.next_request raises an exception 2025-03-15 12:29:37 +01:00
Adrián Chaves 96f3d293cf Refactor the engine loop lightly 2025-03-15 11:52:32 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Adrián Chaves 72c12f10a7 Good Bye, Heartbeat! 2025-03-15 11:14:34 +01:00
Adrián Chaves 1305cfb240 Update test expectations for older versions of Twisted 2025-03-15 03:36:56 +01:00
Adrián Chaves 0a4223ffeb Update test expectations 2025-03-15 02:45:31 +01:00
Adrián Chaves dbdfb1854d test_crawl.py: remove tests covered elsewhere 2025-03-15 02:38:28 +01:00
Adrián Chaves 5c6872a883 Keep pylint happy 2025-03-15 02:32:06 +01:00
Adrián Chaves 734f4e9a1a Cover the new error handling in news.rst and cover it with a test 2025-03-15 02:26:55 +01:00
Adrián Chaves 0a137e40b7 Complete test coverage for exception scenarios of SpiderMiddleware.process_seeds 2025-03-15 00:55:31 +01:00
Adrián Chaves b587b00439 Complete text coverage for exception scenarios of Spider.yield_seeds 2025-03-15 00:26:24 +01:00
Adrián Chaves 8ccb674f33 Log an error but move on if Spider.iter_seeds is not an async generator 2025-03-14 23:54:07 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves 05a21285f4 Address issues reported by static checkers 2025-03-13 15:51:28 +01:00
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves 6ae7666679 Merge branch 'minimal-async-seeds' into seeding-policies 2025-03-13 13:46:59 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 223560dca7 Update the changelog 2025-03-13 11:33:45 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves 57ef3b2689 Update test expectations 2025-03-13 10:12:03 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e6790ec86b Make the idle seeding policy test more reliable 2025-03-13 07:31:25 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves 0ebc057344 Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:31:07 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 6a5425f55f Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:21:36 +01:00
Adrián Chaves 4678a1c7af AsyncIterator → AsyncIterable 2025-03-12 21:09:24 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves cd6fe5fdac Remove partial typing from some code examples to keep them shorter 2025-03-12 17:36:48 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves a8a596b287 Fix direct use of Crawler instead of get_crawler failing with the default reactor 2025-03-12 08:45:07 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves f0c933f094 Implement spider middleware backward compatibility and deprecation warnings 2025-03-12 06:34:11 +01:00
Adrián Chaves 6ce483125f Add backward-compatibility tests for Spider.start_requests 2025-03-12 03:58:35 +01:00