RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,793 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

10793 Commits

Author SHA1 Message Date
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves 6ae7666679 Merge branch 'minimal-async-seeds' into seeding-policies 2025-03-13 13:46:59 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 223560dca7 Update the changelog 2025-03-13 11:33:45 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves 57ef3b2689 Update test expectations 2025-03-13 10:12:03 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e6790ec86b Make the idle seeding policy test more reliable 2025-03-13 07:31:25 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves 0ebc057344 Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:31:07 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 6a5425f55f Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:21:36 +01:00
Adrián Chaves 4678a1c7af AsyncIterator → AsyncIterable 2025-03-12 21:09:24 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves cd6fe5fdac Remove partial typing from some code examples to keep them shorter 2025-03-12 17:36:48 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves a8a596b287 Fix direct use of Crawler instead of get_crawler failing with the default reactor 2025-03-12 08:45:07 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves f0c933f094 Implement spider middleware backward compatibility and deprecation warnings 2025-03-12 06:34:11 +01:00
Adrián Chaves 6ce483125f Add backward-compatibility tests for Spider.start_requests 2025-03-12 03:58:35 +01:00
Adrián Chaves 2ec86ffb42 Address issues reported by pre-commit 2025-03-12 03:20:30 +01:00
Adrián Chaves d4e6b5619a Refactor _next_request for readability 2025-03-12 03:17:34 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 26ecc93228
Run CI only on the main branch, on release branches and on PRs (#6720) 2025-03-11 14:12:11 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves faab15c3f2
Merge pull request #6722 from scrapy/remove-trailing-whitespace 
Remove trailing whitespace
 2025-03-11 12:03:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Adrián Chaves 47bc1422d2 Revert docs change, the warning is not preventing the docs build 2025-03-11 11:43:05 +01:00
Adrián Chaves 1a402578fc Test the current seeding policy, including a complex scenario with async sleeps 2025-03-11 11:07:05 +01:00
Adrián Chaves 30e3ff1d78 Add delay tests for spider middlewares 2025-03-11 01:02:54 +01:00
Adrián Chaves 9d30f5b5ae Minor refactoring 2025-03-11 00:39:46 +01:00
Adrián Chaves 23d7ea8e9d tox.ini: provide a cleaner output by default when targeting specific tests 2025-03-11 00:07:11 +01:00
Adrián Chaves 890f90421b Test a >5s pure-Twisted delay 2025-03-10 10:16:56 +01:00
Andrey Rakhmatullin 2accaa4af4
Merge pull request #6716 from wRAR/better-queuelib-fix 
Better fix for test_non_pickable_object on Windows.
 2025-03-10 13:09:48 +04:00
Adrián Chaves bb68027cce pytest-twisted → twisted.trial (for now) 2025-03-10 09:52:43 +01:00
Adrián Chaves 5fe56cc61e Fix typing 2025-03-10 09:27:00 +01:00
Adrián Chaves 003466496b Fix a >5s asyncio.sleep() raising RuntimeError 2025-03-10 09:16:37 +01:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Adrián Chaves da5ebf0bd7 Avoid duplicate CI runs 2025-03-09 23:52:51 +01:00
Adrián Chaves 5534a5299b Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 20:28:44 +01:00