RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,829 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

2419 Commits

Author SHA1 Message Date
Adrián Chaves c5518bc9fe Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-26 19:45:03 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves 9726538cec Fix expectations for old Twisted 2025-03-26 11:30:13 +01:00
Adrián Chaves fdba9843bb Implement a simple scheduler_empty signal 2025-03-26 11:26:11 +01:00
Adrián Chaves 7b89aeba0b Fix tests 2025-03-26 10:40:27 +01:00
Adrián Chaves 701f582f27 Implement a replacement for the removed test_graceful_crawl_error_handling 2025-03-25 17:51:50 +01:00
Adrián Chaves 1a0b9d4f56 Cover start request sorting examples in the tests 2025-03-25 17:16:05 +01:00
Adrián Chaves 406d2fb100 Update test expectations for the default reactor 2025-03-25 17:06:55 +01:00
Adrián Chaves eddf582ba9 Error out on uncompatible spider middlewares 2025-03-25 16:44:27 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 17d85b8ad7 WIP 2025-03-25 12:52:37 +01:00
Adrián Chaves 5d43d91a95 WIP 2025-03-25 10:59:50 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves b37f3b5422 Fix text replacement artifacts 2025-03-24 11:22:41 +01:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Adrián Chaves 0867ecc0c7 Revert "Support defining Spider.start() as a sync generator" 
This reverts commit debab6cff6.
 2025-03-20 12:09:25 +01:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Adrián Chaves 28b78457aa Update test expectations 2025-03-19 21:33:33 +01:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves debab6cff6 Support defining Spider.start() as a sync generator 2025-03-19 20:16:49 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 57ef3b2689 Update test expectations 2025-03-13 10:12:03 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e6790ec86b Make the idle seeding policy test more reliable 2025-03-13 07:31:25 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves a8a596b287 Fix direct use of Crawler instead of get_crawler failing with the default reactor 2025-03-12 08:45:07 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves f0c933f094 Implement spider middleware backward compatibility and deprecation warnings 2025-03-12 06:34:11 +01:00
Adrián Chaves 6ce483125f Add backward-compatibility tests for Spider.start_requests 2025-03-12 03:58:35 +01:00
Adrián Chaves 2ec86ffb42 Address issues reported by pre-commit 2025-03-12 03:20:30 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 1a402578fc Test the current seeding policy, including a complex scenario with async sleeps 2025-03-11 11:07:05 +01:00
Adrián Chaves 30e3ff1d78 Add delay tests for spider middlewares 2025-03-11 01:02:54 +01:00
Adrián Chaves 9d30f5b5ae Minor refactoring 2025-03-11 00:39:46 +01:00
Adrián Chaves 890f90421b Test a >5s pure-Twisted delay 2025-03-10 10:16:56 +01:00
Adrián Chaves bb68027cce pytest-twisted → twisted.trial (for now) 2025-03-10 09:52:43 +01:00
Adrián Chaves 003466496b Fix a >5s asyncio.sleep() raising RuntimeError 2025-03-10 09:16:37 +01:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Adrián Chaves 5534a5299b Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 20:28:44 +01:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00