32f2aede82
Deprecate non-async counterparts of new async APIs. ( #6979 )
...
* Deprecate non-async counterparts of new async APIs.
* Deprecate Scraper.call_spider().
* Add _schedule_coro().
2025-08-01 00:58:48 +05:00
605e7669a5
Improve the assumption in test_queue_push_pop_priorities(). ( #6974 )
2025-07-28 20:37:45 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list ( #6949 )
2025-07-19 11:54:35 +02:00
e225d0dea4
Ban more imports that import twisted.internet.reactor. ( #6947 )
2025-07-10 21:19:50 +05:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
14eace5d8f
Increase the command timeout in test_short_timeout(). ( #6945 )
2025-07-06 01:56:54 +05:00
4279f2837c
Enable the flake8-builtins rules. ( #6942 )
2025-07-05 21:43:18 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
16f168b406
Deprecate mustbe_deferred() and defer_*(). ( #6937 )
2025-07-04 16:41:15 +05:00
8d82e7bd27
Fix limiting subprocess run time in tests. ( #6933 )
...
* Fix limiting subprocess run time in tests.
* Make test_runspider_dnscache_disabled() faster.
2025-07-01 20:23:35 +05:00
7842180b2f
Merge remote-tracking branch 'origin/2.13'
2025-07-01 17:28:12 +05:00
c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. ( #6911 )
2025-07-01 13:47:43 +02:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. ( #6889 )
2025-06-30 09:19:23 +02:00
db0be1771c
Test refactoring, part 2. ( #6928 )
2025-06-29 23:10:38 +02:00
2c1c10e923
Remove deprecated offsite spider middleware. ( #6926 )
...
* Remove deprecated offsite spider middleware.
* Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:20 +05:00
5fbab843bd
Fix invalid XML samples in tests that break with lxml 6. ( #6919 )
2025-06-27 16:24:09 +05:00
eaae59fbef
Fix invalid XML samples in tests that break with lxml 6. ( #6919 )
2025-06-27 13:05:02 +02:00
ef01a953b1
Fix setting ExecutionEngine._slot to None on close. ( #6910 )
...
* Fix setting ExecutionEngine._slot to None on close.
* Check self._slot in needs_backout().
2025-06-25 17:30:57 +05:00
ff7795b159
Fix setting ExecutionEngine._slot to None on close. ( #6910 )
...
* Fix setting ExecutionEngine._slot to None on close.
* Check self._slot in needs_backout().
2025-06-25 17:29:29 +05:00
d70f8a3f14
Refactoring of test_utils_*. ( #6905 )
2025-06-23 23:39:24 +05:00
7fbd56bc9b
Handle exceptions in _start_request_processing(), cancel it on engine stop ( #6900 )
2025-06-23 22:38:38 +05:00
0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop ( #6900 )
2025-06-23 19:36:54 +02:00
5e20b46e35
Fix FileFeedStorage handling of Windows paths without file:// scheme ( #6897 )
2025-06-23 22:26:04 +05:00
0d86fb69dc
Fix FileFeedStorage handling of Windows paths without file:// scheme ( #6897 )
2025-06-23 18:56:29 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
92c18d15b4
Remove ProcessTest and SiteTest. ( #6885 )
...
* Remove ProcessTest and SiteTest.
* Restore the support for Windows line endings in TestParseCommand.
* Add a test for running a scrapy command in a project subdir.
* Remove pywin32 from test deps.
2025-06-13 22:18:27 +05:00
b4d11b8b25
Further reduce deps on unittest. ( #6884 )
2025-06-11 01:28:09 +02:00
ac956f8595
Replace most of the @inlineCallbacks test helpers. ( #6883 )
2025-06-10 10:02:27 +02:00
24a827c72e
Pin older Twisted in tests, update type hints. ( #6882 )
2025-06-09 13:09:44 +05:00
ba10dcfd1a
Pin older Twisted in tests, update type hints. ( #6882 )
2025-06-09 09:53:42 +02:00
d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. ( #6873 )
2025-06-06 22:59:09 +02:00
d329eedfef
Don't try to close ExecutionEngine.downloader when it doesn't exist. ( #6867 )
2025-06-06 16:26:38 +05:00
657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. ( #6867 )
2025-06-06 13:02:15 +02:00
54474ceb0d
Fix a regression in errback result handling. ( #6863 )
2025-06-06 12:47:38 +05:00
d602f13e8c
Fix a regression in errback result handling. ( #6863 )
2025-06-05 18:02:31 +05:00
8fb8d2c6b8
Add AsyncioLoopingCall. ( #6855 )
2025-06-03 20:21:33 +02:00
3aa5e75787
Use AsyncCrawlerProcess in commands. ( #6845 )
...
* Use AsyncCrawlerProcess in commands.
* Ignore coverage of abstract methods.
* Address feedback.
2025-06-03 17:47:52 +05:00
d400aa3e2d
Add _parallel_asyncio(). ( #6852 )
2025-06-03 11:19:15 +02:00
9cc23641cc
Deprecate _parse_response and implement parse_with_rules ( #6804 )
2025-06-02 17:00:17 +02:00
8ae418df44
Rewrite download handler tests to coroutines. ( #6846 )
2025-06-02 16:02:08 +02:00
a724541a71
Split tests/test_commands.py. ( #6836 )
2025-05-28 21:46:04 +02:00
e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. ( #6824 )
...
* Don't use CrawlerProcess in the commands that don't need it.
* Use a dummy spider loader in runspider.
2025-05-28 19:57:33 +05:00
05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` ( #6817 )
...
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
916fe50974
Fix and ban the top-level twisted.internet.reactor imports. ( #6835 )
2025-05-28 15:53:52 +05:00
dceb85bf3e
Add is_asyncio_available(). ( #6827 )
...
* Add is_asyncio_available().
* Print unexpected warnings in test_install_asyncio_reactor().
* Fix printing warnings.
* Fix printing warnings - 2.
* Skip TestDeferredToFuture on non-asyncio.
* Test the is_asyncio_available() exception.
2025-05-28 14:46:39 +05:00
Andrey Rakhmatullin
Felipe Benevolo
M Ikram Ullah Khan
Laerte Pereira
Thalison Fernandes
Rodrigosnrocha