RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,937 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

10937 Commits

Author SHA1 Message Date
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
Andrey Rakhmatullin 2442536d0f
Add a deepwiki badge, update other badges. (#6793) 2025-05-13 10:22:28 +02:00
Felipe Benevolo 128cb551eb
refactor tests/test_downloadermiddleware_httpcache.py (#6769) 2025-05-12 13:04:34 +02:00
Andrey Rakhmatullin 82a3245158 Bump version: 2.12.0 → 2.13.0 2025-05-08 14:43:34 +05:00
Andrey Rakhmatullin 0f8abd2cce
Merge pull request #6757 from wRAR/relnotes-2.13.0 
Release notes for 2.13.0
 2025-05-08 13:41:56 +04:00
Andrey Rakhmatullin 0ce693dfa9 Update VERSION strings. 2025-05-08 13:23:19 +05:00
Andrey Rakhmatullin b07e6d4ea8 Merge remote-tracking branch 'origin/master' into relnotes-2.13.0 2025-05-08 13:17:47 +05:00
Andrey Rakhmatullin e86b5051c2 Merge commit '373e501' into relnotes-2.13.0 2025-05-08 10:55:57 +05:00
Andrey Rakhmatullin 5f6d1b464b Cover up to 373e501 in the release notes. 2025-05-08 10:55:40 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 373e501f78
Link to scrapy.org from the docs (#6780) 2025-05-07 16:11:22 +02:00
Andrey Rakhmatullin 4899d416e7
Add PyPy 3.11 to CI. (#6697) 2025-05-06 12:31:28 +02:00
Andrey Rakhmatullin 474f8312ff
Merge pull request #6776 from wRAR/deferred-apis 
Add a list of Deferred-only APIs.
 2025-05-05 21:28:50 +04:00
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin 509b572efc
Migrate the build system to hatchling. (#6771) 2025-05-05 09:51:53 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Andrey Rakhmatullin 5dfe7cd7b8
Improve tests for start items. (#6770) 2025-05-05 09:36:52 +02:00
Adrián Chaves 8f059d4095
Merge pull request #6766 from wRAR/errback-tests 
Improve test coverage of Scraper.
 2025-05-02 10:37:28 +02:00
Andrey Rakhmatullin da9078c4bb Add tests for raising CloseSpider in callbacks. 2025-05-01 23:12:39 +05:00
Andrey Rakhmatullin 23c206af35 Improve test coverage of Scraper. 2025-05-01 22:59:18 +05:00
Andrey Rakhmatullin 6deae473d9
Merge pull request #6764 from wRAR/spider-arg-scraper 
Remove the unnecessary spider argument from Scraper and related code.
 2025-04-30 18:33:35 +04:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Andrey Rakhmatullin 4aba7e5f66 Mention the deprecation of TestSpider. 2025-04-25 20:16:26 +05:00
Andrey Rakhmatullin 095140f134 Cover the current master in the release notes (up to daf9db7). 2025-04-25 19:43:29 +05:00
Andrey Rakhmatullin b1f85b5a17 Release notes for 2.13.0, up to b4c2531021. 2025-04-24 20:03:36 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Suejung Shin e50914e0f5
Codecov: Add test analytics (#6741) 2025-03-21 12:28:47 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Andrey Rakhmatullin 8729247213
Fix running simple tests with --reactor=default. (#6735) 2025-03-15 20:33:10 +01:00
Andrey Rakhmatullin 9057bf4e1e
More docs about Deferred<->Future interoperability. (#6734) 2025-03-15 11:47:16 +01:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 26ecc93228
Run CI only on the main branch, on release branches and on PRs (#6720) 2025-03-11 14:12:11 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves faab15c3f2
Merge pull request #6722 from scrapy/remove-trailing-whitespace 
Remove trailing whitespace
 2025-03-11 12:03:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Andrey Rakhmatullin 2accaa4af4
Merge pull request #6716 from wRAR/better-queuelib-fix 
Better fix for test_non_pickable_object on Windows.
 2025-03-10 13:09:48 +04:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00