RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,818 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

2413 Commits

Author SHA1 Message Date
Adrián Chaves 0b9b863bc9 Lower test flakyness 2025-03-17 10:56:47 +01:00
Adrián Chaves 955b40e415 Increase test reliability and min lxml 2025-03-17 10:13:27 +01:00
Adrián Chaves ba88283236 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-15 21:20:10 +01:00
Adrián Chaves dac68c685a Fix test_scheduler_next_request_exception 2025-03-15 21:18:13 +01:00
Adrián Chaves b6707ce3bf Address typing issues 2025-03-15 20:58:16 +01:00
Adrián Chaves a5dc229508 Improve exception handling for Scheduler’s has_pending_requests() and enqueue_request() 2025-03-15 20:50:21 +01:00
Adrián Chaves c8bbe4d968 Log a traceback and assume a None return value if Scheduler.next_request raises an exception 2025-03-15 12:29:37 +01:00
Adrián Chaves 96f3d293cf Refactor the engine loop lightly 2025-03-15 11:52:32 +01:00
Adrián Chaves 72c12f10a7 Good Bye, Heartbeat! 2025-03-15 11:14:34 +01:00
Adrián Chaves 1305cfb240 Update test expectations for older versions of Twisted 2025-03-15 03:36:56 +01:00
Adrián Chaves 0a4223ffeb Update test expectations 2025-03-15 02:45:31 +01:00
Adrián Chaves dbdfb1854d test_crawl.py: remove tests covered elsewhere 2025-03-15 02:38:28 +01:00
Adrián Chaves 5c6872a883 Keep pylint happy 2025-03-15 02:32:06 +01:00
Adrián Chaves 734f4e9a1a Cover the new error handling in news.rst and cover it with a test 2025-03-15 02:26:55 +01:00
Adrián Chaves 0a137e40b7 Complete test coverage for exception scenarios of SpiderMiddleware.process_seeds 2025-03-15 00:55:31 +01:00
Adrián Chaves b587b00439 Complete text coverage for exception scenarios of Spider.yield_seeds 2025-03-15 00:26:24 +01:00
Adrián Chaves 8ccb674f33 Log an error but move on if Spider.iter_seeds is not an async generator 2025-03-14 23:54:07 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves 05a21285f4 Address issues reported by static checkers 2025-03-13 15:51:28 +01:00
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 57ef3b2689 Update test expectations 2025-03-13 10:12:03 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e6790ec86b Make the idle seeding policy test more reliable 2025-03-13 07:31:25 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves a8a596b287 Fix direct use of Crawler instead of get_crawler failing with the default reactor 2025-03-12 08:45:07 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves f0c933f094 Implement spider middleware backward compatibility and deprecation warnings 2025-03-12 06:34:11 +01:00
Adrián Chaves 6ce483125f Add backward-compatibility tests for Spider.start_requests 2025-03-12 03:58:35 +01:00
Adrián Chaves 2ec86ffb42 Address issues reported by pre-commit 2025-03-12 03:20:30 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 1a402578fc Test the current seeding policy, including a complex scenario with async sleeps 2025-03-11 11:07:05 +01:00
Adrián Chaves 30e3ff1d78 Add delay tests for spider middlewares 2025-03-11 01:02:54 +01:00
Adrián Chaves 9d30f5b5ae Minor refactoring 2025-03-11 00:39:46 +01:00
Adrián Chaves 890f90421b Test a >5s pure-Twisted delay 2025-03-10 10:16:56 +01:00
Adrián Chaves bb68027cce pytest-twisted → twisted.trial (for now) 2025-03-10 09:52:43 +01:00
Adrián Chaves 003466496b Fix a >5s asyncio.sleep() raising RuntimeError 2025-03-10 09:16:37 +01:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Adrián Chaves 5534a5299b Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 20:28:44 +01:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
Andrey Rakhmatullin 02ed71d887
Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
Mehraz Hossain Rumman 044c3f69ed
Deprecate InitSpider (#6714) 2025-03-09 20:18:57 +01:00
Adrián Chaves 3d93441ea6 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 09:47:20 +01:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Andrey Rakhmatullin 1469b2739e
Drop tests/test_loader_deprecated.py. (#6704) 2025-03-08 17:50:54 +01:00