RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,842 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2821 Commits

Author SHA1 Message Date
Adrián Chaves 5e18a132da Clean up from self-review 2025-03-27 14:03:39 +01:00
Adrián Chaves 6a378bb94a Remove enum-tools 2025-03-27 12:05:20 +01:00
Adrián Chaves c5518bc9fe Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-26 19:45:03 +01:00
Adrián Chaves bfb720764b Restore old issue title reference 2025-03-26 19:44:29 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves fdba9843bb Implement a simple scheduler_empty signal 2025-03-26 11:26:11 +01:00
Adrián Chaves 97a0fed020 Document the execution order change of non-generator start_requests, for the sake of completion 2025-03-25 17:27:58 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 17d85b8ad7 WIP 2025-03-25 12:52:37 +01:00
Adrián Chaves 52f931088c partial coroutine support → coroutine support 2025-03-25 11:02:32 +01:00
Adrián Chaves 5d43d91a95 WIP 2025-03-25 10:59:50 +01:00
Adrián Chaves 68f63e1430 Fix bad ports from the main PR 2025-03-24 17:29:04 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves cc49ea9f18 Inclued () in …start_requests mentions on news.rst 2025-03-24 10:58:19 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves f91723628a Disambiguate method refenrece in the docs 2025-03-19 20:27:38 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves 223560dca7 Update the changelog 2025-03-13 11:33:45 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 4678a1c7af AsyncIterator → AsyncIterable 2025-03-12 21:09:24 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves cd6fe5fdac Remove partial typing from some code examples to keep them shorter 2025-03-12 17:36:48 +01:00
Adrián Chaves e049768987 Remove seeding policy references for a minimal implementation 2025-03-12 09:53:08 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Adrián Chaves 47bc1422d2 Revert docs change, the warning is not preventing the docs build 2025-03-11 11:43:05 +01:00
Adrián Chaves 3d93441ea6 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 09:47:20 +01:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin c200458f24
Add more docs for updating sync spider middlewares. (#6688) 2025-02-25 00:16:23 +01:00