RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,806 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

10806 Commits

Author SHA1 Message Date
Adrián Chaves b37f3b5422 Fix text replacement artifacts 2025-03-24 11:22:41 +01:00
Adrián Chaves cc49ea9f18 Inclued () in …start_requests mentions on news.rst 2025-03-24 10:58:19 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Adrián Chaves 0867ecc0c7 Revert "Support defining Spider.start() as a sync generator" 
This reverts commit debab6cff6.
 2025-03-20 12:09:25 +01:00
Adrián Chaves 28b78457aa Update test expectations 2025-03-19 21:33:33 +01:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves f91723628a Disambiguate method refenrece in the docs 2025-03-19 20:27:38 +01:00
Adrián Chaves dfc7cd58e7 Fix issue reported by mypy 2025-03-19 20:19:14 +01:00
Adrián Chaves debab6cff6 Support defining Spider.start() as a sync generator 2025-03-19 20:16:49 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Andrey Rakhmatullin 8729247213
Fix running simple tests with --reactor=default. (#6735) 2025-03-15 20:33:10 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves 6ae7666679 Merge branch 'minimal-async-seeds' into seeding-policies 2025-03-13 13:46:59 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 223560dca7 Update the changelog 2025-03-13 11:33:45 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves 57ef3b2689 Update test expectations 2025-03-13 10:12:03 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves e6790ec86b Make the idle seeding policy test more reliable 2025-03-13 07:31:25 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves 0ebc057344 Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:31:07 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 6a5425f55f Replace RuntimeError handling with Andrey’s lock-based approach 2025-03-12 21:21:36 +01:00
Adrián Chaves 4678a1c7af AsyncIterator → AsyncIterable 2025-03-12 21:09:24 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves cd6fe5fdac Remove partial typing from some code examples to keep them shorter 2025-03-12 17:36:48 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves a8a596b287 Fix direct use of Crawler instead of get_crawler failing with the default reactor 2025-03-12 08:45:07 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves f0c933f094 Implement spider middleware backward compatibility and deprecation warnings 2025-03-12 06:34:11 +01:00
Adrián Chaves 6ce483125f Add backward-compatibility tests for Spider.start_requests 2025-03-12 03:58:35 +01:00
Adrián Chaves 2ec86ffb42 Address issues reported by pre-commit 2025-03-12 03:20:30 +01:00
Adrián Chaves d4e6b5619a Refactor _next_request for readability 2025-03-12 03:17:34 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 26ecc93228
Run CI only on the main branch, on release branches and on PRs (#6720) 2025-03-11 14:12:11 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves faab15c3f2
Merge pull request #6722 from scrapy/remove-trailing-whitespace 
Remove trailing whitespace
 2025-03-11 12:03:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00