RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,876 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

2454 Commits

Author SHA1 Message Date
Adrián Chaves a04e85e08b Re-implement front-load support 2025-03-31 11:49:28 +02:00
Adrián Chaves 43e789f776 Use self 2025-03-31 10:02:31 +02:00
Adrián Chaves 9417de1df9 Review uses of seed 2025-03-31 09:53:52 +02:00
Adrián Chaves e984326777 Improve test reliability 2025-03-31 09:52:34 +02:00
Adrián Chaves a9ce836f98 Fix test expectations and improve reliability 2025-03-28 16:20:15 +01:00
Adrián Chaves 85e3dc13f1 Merge branch 'minimal-async-seeds' into async-seeds 2025-03-28 15:54:23 +01:00
Adrián Chaves 2a148d07b0 Increase coverage 2025-03-27 15:51:08 +01:00
Adrián Chaves 40adc6a356 Increase coverage 2025-03-27 15:39:37 +01:00
Adrián Chaves e360a4cc98 Increase coverage 2025-03-27 15:24:57 +01:00
Adrián Chaves 8057f06b39 Increase coverage 2025-03-27 15:16:02 +01:00
Adrián Chaves c67edb78b9 Deprecate calls to Spider.start_requests 2025-03-27 15:04:34 +01:00
Adrián Chaves 5e18a132da Clean up from self-review 2025-03-27 14:03:39 +01:00
Adrián Chaves 581c969ead Remove tests for undefined behavior 2025-03-27 00:11:30 +01:00
Adrián Chaves 49b839b270 Reliability improvements 2025-03-26 23:28:29 +01:00
Adrián Chaves 476234b07a Fix tests 2025-03-26 22:28:02 +01:00
Adrián Chaves c5518bc9fe Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-26 19:45:03 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves 9726538cec Fix expectations for old Twisted 2025-03-26 11:30:13 +01:00
Adrián Chaves fdba9843bb Implement a simple scheduler_empty signal 2025-03-26 11:26:11 +01:00
Adrián Chaves 7b89aeba0b Fix tests 2025-03-26 10:40:27 +01:00
Adrián Chaves 701f582f27 Implement a replacement for the removed test_graceful_crawl_error_handling 2025-03-25 17:51:50 +01:00
Adrián Chaves 1a0b9d4f56 Cover start request sorting examples in the tests 2025-03-25 17:16:05 +01:00
Adrián Chaves 406d2fb100 Update test expectations for the default reactor 2025-03-25 17:06:55 +01:00
Adrián Chaves eddf582ba9 Error out on uncompatible spider middlewares 2025-03-25 16:44:27 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 17d85b8ad7 WIP 2025-03-25 12:52:37 +01:00
Adrián Chaves 5d43d91a95 WIP 2025-03-25 10:59:50 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves b37f3b5422 Fix text replacement artifacts 2025-03-24 11:22:41 +01:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Adrián Chaves 0867ecc0c7 Revert "Support defining Spider.start() as a sync generator" 
This reverts commit debab6cff6.
 2025-03-20 12:09:25 +01:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Adrián Chaves 28b78457aa Update test expectations 2025-03-19 21:33:33 +01:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves 220c70bc0c Improve test reliability 2025-03-19 20:25:55 +01:00
Adrián Chaves debab6cff6 Support defining Spider.start() as a sync generator 2025-03-19 20:16:49 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Adrián Chaves 0b9b863bc9 Lower test flakyness 2025-03-17 10:56:47 +01:00
Adrián Chaves 955b40e415 Increase test reliability and min lxml 2025-03-17 10:13:27 +01:00
Adrián Chaves ba88283236 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-15 21:20:10 +01:00
Adrián Chaves dac68c685a Fix test_scheduler_next_request_exception 2025-03-15 21:18:13 +01:00
Adrián Chaves b6707ce3bf Address typing issues 2025-03-15 20:58:16 +01:00
Adrián Chaves a5dc229508 Improve exception handling for Scheduler’s has_pending_requests() and enqueue_request() 2025-03-15 20:50:21 +01:00
Adrián Chaves c8bbe4d968 Log a traceback and assume a None return value if Scheduler.next_request raises an exception 2025-03-15 12:29:37 +01:00
Adrián Chaves 96f3d293cf Refactor the engine loop lightly 2025-03-15 11:52:32 +01:00
Adrián Chaves 72c12f10a7 Good Bye, Heartbeat! 2025-03-15 11:14:34 +01:00
Adrián Chaves 1305cfb240 Update test expectations for older versions of Twisted 2025-03-15 03:36:56 +01:00