RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,889 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2436 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 14eace5d8f
Increase the command timeout in test_short_timeout(). (#6945) 2025-07-06 01:56:54 +05:00
Andrey Rakhmatullin 4279f2837c
Enable the flake8-builtins rules. (#6942) 2025-07-05 21:43:18 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 16f168b406
Deprecate mustbe_deferred() and defer_*(). (#6937) 2025-07-04 16:41:15 +05:00
Andrey Rakhmatullin 8d82e7bd27
Fix limiting subprocess run time in tests. (#6933) 
* Fix limiting subprocess run time in tests.

* Make test_runspider_dnscache_disabled() faster.
 2025-07-01 20:23:35 +05:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Laerte Pereira dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting (#6921) 
* Deprecate CONCURRENT_REQUESTS_PER_IP setting

* Typing fix

* PR review

* tests

* oops

* Update scrapy/settings/default_settings.py

Co-authored-by: Adrián Chaves <adrian@chaves.gal>

* update logic

---------

Co-authored-by: Adrián Chaves <adrian@chaves.gal>
 2025-06-30 16:51:05 +05:00
Andrey Rakhmatullin 3843091c5f
Mark abstract base classes as such. (#6930) 
* Mark abstract base classes as such.

* Remove an unneeded method.

* Fix exporter test coverage.
 2025-06-30 15:14:34 +05:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin db0be1771c
Test refactoring, part 2. (#6928) 2025-06-29 23:10:38 +02:00
Andrey Rakhmatullin 2c1c10e923
Remove deprecated offsite spider middleware. (#6926) 
* Remove deprecated offsite spider middleware.

* Add a deprecation notice for the offsite spider middleware.
 2025-06-29 02:35:20 +05:00
Andrey Rakhmatullin 5fbab843bd Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 16:24:09 +05:00
Andrey Rakhmatullin eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. (#6919) 2025-06-27 13:05:02 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910) 
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
 2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin d70f8a3f14
Refactoring of test_utils_*. (#6905) 2025-06-23 23:39:24 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Thalison Fernandes 5e20b46e35 Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 22:26:04 +05:00
Thalison Fernandes 0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme (#6897) 2025-06-23 18:56:29 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin b4d11b8b25
Further reduce deps on unittest. (#6884) 2025-06-11 01:28:09 +02:00
Andrey Rakhmatullin ac956f8595
Replace most of the @inlineCallbacks test helpers. (#6883) 2025-06-10 10:02:27 +02:00
Andrey Rakhmatullin 24a827c72e Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 13:09:44 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin d329eedfef Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 16:26:38 +05:00
Andrey Rakhmatullin 657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 13:02:15 +02:00
Andrey Rakhmatullin 54474ceb0d Fix a regression in errback result handling. (#6863) 2025-06-06 12:47:38 +05:00
Andrey Rakhmatullin d602f13e8c
Fix a regression in errback result handling. (#6863) 2025-06-05 18:02:31 +05:00
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 3aa5e75787
Use AsyncCrawlerProcess in commands. (#6845) 
* Use AsyncCrawlerProcess in commands.

* Ignore coverage of abstract methods.

* Address feedback.
 2025-06-03 17:47:52 +05:00
Andrey Rakhmatullin d400aa3e2d
Add _parallel_asyncio(). (#6852) 2025-06-03 11:19:15 +02:00
Rodrigosnrocha 9cc23641cc
Deprecate _parse_response and implement parse_with_rules (#6804) 2025-06-02 17:00:17 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Andrey Rakhmatullin a724541a71
Split tests/test_commands.py. (#6836) 2025-05-28 21:46:04 +02:00
Andrey Rakhmatullin e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. (#6824) 
* Don't use CrawlerProcess in the commands that don't need it.

* Use a dummy spider loader in runspider.
 2025-05-28 19:57:33 +05:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817) 
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
 2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin dceb85bf3e
Add is_asyncio_available(). (#6827) 
* Add is_asyncio_available().

* Print unexpected warnings in test_install_asyncio_reactor().

* Fix printing warnings.

* Fix printing warnings - 2.

* Skip TestDeferredToFuture on non-asyncio.

* Test the is_asyncio_available() exception.
 2025-05-28 14:46:39 +05:00
Adrián Chaves f98ffc71d2
2.13.1 (#6832) 2025-05-28 10:58:34 +02:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821) 
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
 2025-05-27 18:19:47 +05:00
Adrián Chaves e3f82afaf1 Add a test for ScrapyPriorityQueue pop order 2025-05-27 10:01:00 +02:00
Adrián Chaves 9d92d16510 Prioritize other requests over start requests 2025-05-27 08:32:24 +02:00
Andrey Rakhmatullin b9caaf8a63 Simplify deferred_from_coro(), add more tests. 2025-05-16 17:16:45 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin 82acef3051
Add AsyncCrawlerRunner. (#6796) 2025-05-14 16:21:18 +02:00
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00