RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,812 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

2402 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin d400aa3e2d
Add _parallel_asyncio(). (#6852) 2025-06-03 11:19:15 +02:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Andrey Rakhmatullin a724541a71
Split tests/test_commands.py. (#6836) 2025-05-28 21:46:04 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821) 
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
 2025-05-27 18:19:47 +05:00
Andrey Rakhmatullin b9caaf8a63 Simplify deferred_from_coro(), add more tests. 2025-05-16 17:16:45 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
Felipe Benevolo 128cb551eb
refactor tests/test_downloadermiddleware_httpcache.py (#6769) 2025-05-12 13:04:34 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 2a1edbd473
Remove usages of TestCase._wait(). (#6773) 2025-05-05 09:44:17 +02:00
Andrey Rakhmatullin ff1ac75c9e
Fix shutdown tests. (#6772) 2025-05-05 09:37:38 +02:00
Andrey Rakhmatullin 5dfe7cd7b8
Improve tests for start items. (#6770) 2025-05-05 09:36:52 +02:00
Andrey Rakhmatullin da9078c4bb Add tests for raising CloseSpider in callbacks. 2025-05-01 23:12:39 +05:00
Andrey Rakhmatullin 23c206af35 Improve test coverage of Scraper. 2025-05-01 22:59:18 +05:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Andrey Rakhmatullin daf9db72b2
Base class for universal spider middlewares (#6693) 
* Initial BaseSpiderMiddleware.

* Rename the new methods.

* Remove the spider argument from new BaseSpiderMiddleware methods.

* Add docs for BaseSpiderMiddleware.

* Silence pylint.

* Add BaseSpiderMiddleware tests.

* Add a release note.
 2025-04-23 19:29:04 +05:00
Andrey Rakhmatullin 9f99da8f86
Convert test_downloadermiddleware_robotstxt.py from callbacks to awaits. (#6743) 2025-03-24 09:26:25 +01:00
Mehraz Hossain Rumman 3ca882fba8
Syntax Error Fixed (#6738) 
* Syntax error fix issue #6731

* test case added

* extra logic removed

* mock spider fixture

* Update scrapy/utils/misc.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* settings.rst updated

* settings.rst updated

* settings.rst updated

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-03-20 17:02:10 +05:00
Laerte Pereira 2ee01efe49
feat: Add count to spider_exceptions stats (#6740) 
* feat: Add overall exception_count to spider_exceptions stats

* Remove variable

* Update test_closespider.py

* Update test_closespider.py

* Rename exception_count → count
 2025-03-20 15:49:44 +05:00
Andrey Rakhmatullin fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). (#6732) 2025-03-14 20:46:17 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. (#6725) 2025-03-11 18:00:36 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Andrey Rakhmatullin 0bbfca6c1d Better fix for test_non_pickable_object on Windows. 2025-03-10 13:15:28 +05:00
Andrey Rakhmatullin 7bbe775040
Converting tests to plain asserts, part 5. (#6712) 2025-03-09 20:24:45 +01:00
Andrey Rakhmatullin d442227fa7
Converting tests to plain asserts, part 8. (#6711) 2025-03-09 20:24:12 +01:00
Andrey Rakhmatullin 380c2279b9
Converting tests to plain asserts, part 7. (#6710) 2025-03-09 20:23:51 +01:00
Andrey Rakhmatullin 02ed71d887
Converting tests to plain asserts, part 6. (#6709) 2025-03-09 20:20:24 +01:00
Mehraz Hossain Rumman 044c3f69ed
Deprecate InitSpider (#6714) 2025-03-09 20:18:57 +01:00
Andrey Rakhmatullin 1469b2739e
Drop tests/test_loader_deprecated.py. (#6704) 2025-03-08 17:50:54 +01:00
Andrey Rakhmatullin 40833afc86
Work around a queuelib test file close problem. (#6703) 2025-03-07 19:33:41 +01:00
Andrey Rakhmatullin 3ded1dfe31
Converting tests to plain asserts, part 4. (#6702) 2025-03-07 17:25:15 +01:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 35212ec5b0
Merge pull request #6696 from wRAR/deprecate-test-utils 
Move most of the test utils inside tests.
 2025-03-05 22:07:15 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin a5731c1944 Move most of the test utils inside tests. 2025-03-02 21:04:12 +05:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 391af6afcc
Unknown encoding handling in HttpCompressionMiddleware, restore x-gzip support (#6618) 
* Unknown encoding handling in HttpCompressionMiddleware.

* Implement the changes for unknown encoding handling.

* Restore support for Content-Encoding: x-gzip.

* Simplify the decoding logic.

* Add tests for the unsupported encoding warning.

* Add a test for the "no zstandard" warning.
 2025-02-27 22:37:01 +05:00
Matt Winter 8c34e6d9a4
curl: add support for parsing -b,--cookie (#6684) 2025-02-19 10:17:37 +01:00