RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,876 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

1955 Commits

Author SHA1 Message Date
Adrián Chaves a04e85e08b Re-implement front-load support 2025-03-31 11:49:28 +02:00
Adrián Chaves 27ae12dc59 Fix method reference in the docs 2025-03-31 10:05:34 +02:00
Adrián Chaves 9417de1df9 Review uses of seed 2025-03-31 09:53:52 +02:00
Adrián Chaves 85e3dc13f1 Merge branch 'minimal-async-seeds' into async-seeds 2025-03-28 15:54:23 +01:00
Adrián Chaves 5e18a132da Clean up from self-review 2025-03-27 14:03:39 +01:00
Adrián Chaves c5518bc9fe Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-26 19:45:03 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves fdba9843bb Implement a simple scheduler_empty signal 2025-03-26 11:26:11 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 17d85b8ad7 WIP 2025-03-25 12:52:37 +01:00
Adrián Chaves 52f931088c partial coroutine support → coroutine support 2025-03-25 11:02:32 +01:00
Adrián Chaves 5d43d91a95 WIP 2025-03-25 10:59:50 +01:00
Adrián Chaves 68f63e1430 Fix bad ports from the main PR 2025-03-24 17:29:04 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves f91723628a Disambiguate method refenrece in the docs 2025-03-19 20:27:38 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Adrián Chaves 193f170df9 Minor refactoring 2025-03-19 19:31:02 +01:00
Adrián Chaves 1f65d244e5 Finish the page on optimizations 2025-03-17 12:26:19 +01:00
Adrián Chaves 28fa70a969 WIP documentation improvements 2025-03-17 01:12:26 +01:00
Adrián Chaves 65f3f0d208 WIP documentation improvements 2025-03-16 00:32:23 +01:00
Adrián Chaves ba88283236 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-15 21:20:10 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Adrián Chaves 8ccb674f33 Log an error but move on if Spider.iter_seeds is not an async generator 2025-03-14 23:54:07 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 4678a1c7af AsyncIterator → AsyncIterable 2025-03-12 21:09:24 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves cd6fe5fdac Remove partial typing from some code examples to keep them shorter 2025-03-12 17:36:48 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Adrián Chaves 3d93441ea6 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 09:47:20 +01:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00