RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,812 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

10812 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin d400aa3e2d
Add _parallel_asyncio(). (#6852) 2025-06-03 11:19:15 +02:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Adrián Chaves 8f92a26636
Avoid raw HTML in the README (#6839) 2025-05-30 09:33:17 +02:00
Andrey Rakhmatullin a724541a71
Split tests/test_commands.py. (#6836) 2025-05-28 21:46:04 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves c480c77f54
Shorten the README and remove broken links to scrapy.org (#6833) 2025-05-28 11:35:18 +02:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Adrián Chaves 7b4cf06b6e
Feature the new logo in the README (#6831) 2025-05-28 10:54:36 +02:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821) 
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
 2025-05-27 18:19:47 +05:00
Andrey Rakhmatullin 816d23da30
Make the release notes work better on PyPI. (#6826) 2025-05-26 21:31:28 +02:00
Andrey Rakhmatullin f2fc177f1f
Fix a wrong versionadded usage. (#6822) 2025-05-23 11:06:33 +02:00
Keval Sakhiya ff7d29654a
Fix typo in documentation and code: 'needs_backoff' -> 'needs_backout' (#6815) 
Corrected the typo in the code and documentation where 'needs_backoff' was incorrectly used instead of 'needs_backout'.
 2025-05-20 07:51:31 +02:00
Andrey Rakhmatullin 4b9043b532
Merge pull request #6803 from wRAR/simplify-defer 
Simplify deferred_from_coro(), add more tests.
 2025-05-16 18:11:56 +04:00
Andrey Rakhmatullin b9caaf8a63 Simplify deferred_from_coro(), add more tests. 2025-05-16 17:16:45 +05:00
Andrey Rakhmatullin 457ba39719
Merge pull request #6801 from wRAR/signals-async 
Add `send_catch_log_async()`.
 2025-05-15 21:51:26 +04:00
Andrey Rakhmatullin 3c2cd53abb Skip the doctest. 2025-05-15 22:17:37 +05:00
Andrey Rakhmatullin bf1bfaaa3e Slight improvements for the signal docs. 2025-05-15 20:02:38 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
Andrey Rakhmatullin 2442536d0f
Add a deepwiki badge, update other badges. (#6793) 2025-05-13 10:22:28 +02:00
Felipe Benevolo 128cb551eb
refactor tests/test_downloadermiddleware_httpcache.py (#6769) 2025-05-12 13:04:34 +02:00
Andrey Rakhmatullin 82a3245158 Bump version: 2.12.0 → 2.13.0 2025-05-08 14:43:34 +05:00
Andrey Rakhmatullin 0f8abd2cce
Merge pull request #6757 from wRAR/relnotes-2.13.0 
Release notes for 2.13.0
 2025-05-08 13:41:56 +04:00
Andrey Rakhmatullin 0ce693dfa9 Update VERSION strings. 2025-05-08 13:23:19 +05:00
Andrey Rakhmatullin b07e6d4ea8 Merge remote-tracking branch 'origin/master' into relnotes-2.13.0 2025-05-08 13:17:47 +05:00
Andrey Rakhmatullin e86b5051c2 Merge commit '373e501' into relnotes-2.13.0 2025-05-08 10:55:57 +05:00
Andrey Rakhmatullin 5f6d1b464b Cover up to 373e501 in the release notes. 2025-05-08 10:55:40 +05:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves 373e501f78
Link to scrapy.org from the docs (#6780) 2025-05-07 16:11:22 +02:00
Andrey Rakhmatullin 4899d416e7
Add PyPy 3.11 to CI. (#6697) 2025-05-06 12:31:28 +02:00
Andrey Rakhmatullin 474f8312ff
Merge pull request #6776 from wRAR/deferred-apis 
Add a list of Deferred-only APIs.
 2025-05-05 21:28:50 +04:00
Andrey Rakhmatullin acb5f895cd
Update docs/topics/coroutines.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2025-05-05 22:28:36 +05:00
Adrián Chaves 523fc25c4d
Document default values set by startproject (#6775) 2025-05-05 18:51:15 +02:00
Andrey Rakhmatullin b93290f28a Add a list of Deferred-only APIs. 2025-05-05 19:38:04 +05:00
Andrey Rakhmatullin 509b572efc
Migrate the build system to hatchling. (#6771) 2025-05-05 09:51:53 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Andrey Rakhmatullin 5dfe7cd7b8
Improve tests for start items. (#6770) 2025-05-05 09:36:52 +02:00
Adrián Chaves 8f059d4095
Merge pull request #6766 from wRAR/errback-tests 
Improve test coverage of Scraper.
 2025-05-02 10:37:28 +02:00
Andrey Rakhmatullin da9078c4bb Add tests for raising CloseSpider in callbacks. 2025-05-01 23:12:39 +05:00
Andrey Rakhmatullin 23c206af35 Improve test coverage of Scraper. 2025-05-01 22:59:18 +05:00
Andrey Rakhmatullin 6deae473d9
Merge pull request #6764 from wRAR/spider-arg-scraper 
Remove the unnecessary spider argument from Scraper and related code.
 2025-04-30 18:33:35 +04:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00