RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,909 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

2848 Commits

Author SHA1 Message Date
Adrián Chaves 51c39308d8 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-05-07 21:16:13 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 373e501f78
Link to scrapy.org from the docs (#6780) 2025-05-07 16:11:22 +02:00
Andrey Rakhmatullin 474f8312ff
Merge pull request #6776 from wRAR/deferred-apis 
Add a list of Deferred-only APIs.
 2025-05-05 21:28:50 +04:00
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Adrián Chaves 76ccc50622 Add an example of higher-priority leaf requests 2025-04-01 13:16:14 +02:00
Adrián Chaves 9dc9999eaa Add docs about universal start request definition 2025-04-01 12:13:59 +02:00
Adrián Chaves 6c1852d313 Update the FAQ 2025-04-01 11:07:20 +02:00
Adrián Chaves 28df50ddcc Minor edits around Optimization 2025-03-31 17:13:40 +02:00
Adrián Chaves 5c9a931e8b Re-implement idle support 2025-03-31 15:18:38 +02:00
Adrián Chaves a04e85e08b Re-implement front-load support 2025-03-31 11:49:28 +02:00
Adrián Chaves 27ae12dc59 Fix method reference in the docs 2025-03-31 10:05:34 +02:00
Adrián Chaves 9417de1df9 Review uses of seed 2025-03-31 09:53:52 +02:00
Adrián Chaves 85e3dc13f1 Merge branch 'minimal-async-seeds' into async-seeds 2025-03-28 15:54:23 +01:00
Adrián Chaves 5e18a132da Clean up from self-review 2025-03-27 14:03:39 +01:00
Adrián Chaves 6a378bb94a Remove enum-tools 2025-03-27 12:05:20 +01:00
Adrián Chaves c5518bc9fe Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-26 19:45:03 +01:00
Adrián Chaves bfb720764b Restore old issue title reference 2025-03-26 19:44:29 +01:00
Adrián Chaves 313f9de28d Implement lazy support 2025-03-26 19:40:53 +01:00
Adrián Chaves fdba9843bb Implement a simple scheduler_empty signal 2025-03-26 11:26:11 +01:00
Adrián Chaves 97a0fed020 Document the execution order change of non-generator start_requests, for the sake of completion 2025-03-25 17:27:58 +01:00
Adrián Chaves 19c910f731 Complete the new loop implementation (no lazy support yet) 2025-03-25 15:28:21 +01:00
Adrián Chaves 17d85b8ad7 WIP 2025-03-25 12:52:37 +01:00
Adrián Chaves 52f931088c partial coroutine support → coroutine support 2025-03-25 11:02:32 +01:00
Adrián Chaves 5d43d91a95 WIP 2025-03-25 10:59:50 +01:00
Adrián Chaves 68f63e1430 Fix bad ports from the main PR 2025-03-24 17:29:04 +01:00
Adrián Chaves 4ac921f389 WIP 2025-03-24 17:26:58 +01:00
Adrián Chaves cc49ea9f18 Inclued () in …start_requests mentions on news.rst 2025-03-24 10:58:19 +01:00
Adrián Chaves 8347084174 Merge branch 'seeding-policies' into minimal-async-seeds 2025-03-20 15:34:19 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Adrián Chaves 07b6fafdd0 Rename the recommended name for the process_start 1st parameter (seeds → start) 2025-03-19 20:53:02 +01:00
Adrián Chaves f91723628a Disambiguate method refenrece in the docs 2025-03-19 20:27:38 +01:00
Adrián Chaves ea3e6d2e42 yield_seeds, process_seeds → start, process_start 2025-03-19 19:42:13 +01:00
Adrián Chaves 3193d17b61 Merge remote-tracking branch 'scrapy/master' into minimal-async-seeds 2025-03-19 19:33:14 +01:00
Adrián Chaves 193f170df9 Minor refactoring 2025-03-19 19:31:02 +01:00
Adrián Chaves 1f65d244e5 Finish the page on optimizations 2025-03-17 12:26:19 +01:00
Adrián Chaves 955b40e415 Increase test reliability and min lxml 2025-03-17 10:13:27 +01:00
Adrián Chaves 28fa70a969 WIP documentation improvements 2025-03-17 01:12:26 +01:00
Adrián Chaves 65f3f0d208 WIP documentation improvements 2025-03-16 00:32:23 +01:00
Adrián Chaves ba88283236 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-15 21:20:10 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Adrián Chaves 734f4e9a1a Cover the new error handling in news.rst and cover it with a test 2025-03-15 02:26:55 +01:00
Adrián Chaves 8ccb674f33 Log an error but move on if Spider.iter_seeds is not an async generator 2025-03-14 23:54:07 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Adrián Chaves 05a21285f4 Address issues reported by static checkers 2025-03-13 15:51:28 +01:00
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00