61f99f2df1
More granular untyped defs checking in tests. ( #7712 )
2026-07-08 10:05:45 +02:00
361f689df7
Improve test coverage for crawler.py ( #7682 )
...
* Improve test coverage for crawler.py
* Silence mypy warnings
* Improve test coverage for crawler.py
2026-07-01 11:53:07 +05:00
b6596de317
Do not ignore CrawlerProcess settings ( #7647 )
...
* Do not ignore CrawlerProcess settings
* Update test_crawlerrunner_accepts_crawler
2026-06-22 20:10:10 +05:00
c9f952c258
Refactor and improve catching warnings in tests. ( #7643 )
2026-06-19 21:04:34 +02:00
3a36955261
Assorted test fixes ( #7616 )
2026-06-15 08:39:58 +02:00
d59f9b644a
Replace _SettingsKeyT with str. ( #7586 )
2026-06-10 11:12:53 +02:00
ddafb37a7c
Assorted test fixes ( #7585 )
...
* Fix the mitmproxy junitxml file name.
* Fix TestAsyncCrawlerRunnerHasSpider regression.
* Update split out file refs.
* Update the test_counter_handler() docstring.
2026-06-10 11:01:06 +02:00
fed75a6c76
Refactor test_crawl.py and test_crawler.py. ( #7566 )
2026-06-03 14:48:12 +02:00
b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. ( #7394 )
2026-04-02 10:20:54 +02:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
939db88b04
Disable mypy allow_untyped_calls. ( #7331 )
2026-03-24 10:31:17 +01:00
584d99af30
Split long test files ( #7329 )
2026-03-13 10:30:25 +01:00
abd025f78e
Pass correct env to PopenSpawn() in tests. ( #7279 )
2026-02-23 11:34:56 +01:00
11977afba5
Rename async test decorators. ( #7234 )
2026-02-02 20:39:43 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
4e1faf883d
Foundations for the reactorless mode. ( #7199 )
...
* Foundations for the reactorless mode.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
2026-01-31 20:48:54 +05:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
a2463325db
Allow Crawler.crawl_async() without asyncio. ( #7202 )
2025-12-31 18:33:59 +01:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
10850e7d29
Fix extra stdout output in tests ( #7070 )
2025-10-06 11:19:34 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
57f539d7a5
Deprecate Crawler.stop(). ( #6999 )
2025-08-11 08:32:55 +02:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
e0b9f2d8f6
Don't use CrawlerProcess in the commands that don't need it. ( #6824 )
...
* Don't use CrawlerProcess in the commands that don't need it.
* Use a dummy spider loader in runspider.
2025-05-28 19:57:33 +05:00
05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` ( #6817 )
...
* Add a basic Crawler.crawl_async().
* Add custom loop tests for *CrawlerRunner.
* Add AsyncCrawlerProcess.
* Update related docs.
* Update practices.rst.
* Address test failures.
* Add a note about AsyncCrawler* to the docs about switching reactors.
* Address feedback.
* Update for TID253.
* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
82acef3051
Add AsyncCrawlerRunner. ( #6796 )
2025-05-14 16:21:18 +02:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
ff1ac75c9e
Fix shutdown tests. ( #6772 )
2025-05-05 09:37:38 +02:00
fc566a7ff9
Don't install the reactor in is_asyncio_reactor_installed(). ( #6732 )
2025-03-14 20:46:17 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
5a605969bd
Converting tests to plain asserts, part 2. ( #6699 )
2025-03-06 20:52:41 +01:00
8d92c28a16
Switch to pytest.raises(). ( #6680 )
...
* Switch to pytest.raises().
* Add matches= to broad pytest.raises().
* Adjust the test_nonserializable_object() regex for Python <= 3.11.
* Adjust the test_nonserializable_object() regex for PyPy.
* Adjust other test exception regexes for PyPy.
* Cleanup.
2025-02-27 23:13:04 +05:00
a898331d14
Preparations for switching to direct pytest. ( #6678 )
2025-02-16 20:28:58 +01:00
273620488c
Enable PTH Ruff rules.
2025-01-01 22:21:51 +05:00
838ff99f37
Enable RUF Ruff rules.
2025-01-01 21:41:08 +05:00
21b9ba717c
Allow customizing logged software versions ( #6582 )
...
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-12-16 14:46:23 +01:00
8c23da943c
Integrating configs into pyproject.toml ( #6547 )
2024-11-19 19:21:15 +05:00
b042ad255d
Address some previously ignored pylint messages. ( #6531 )
2024-11-11 11:49:52 +01:00
d85c39f5bc
Deprecation removals. ( #6500 )
...
* Deprecation removals.
* Clean up the default pytest filterwarnings.
* Remove test_get_images_old().
* Redo boto-requiring test filtering.
* Remove an unused function.
* Improve the Crawler.crawl() error message.
* Fix the test.
2024-10-31 18:06:22 +05:00
04d0411bf7
Filter test-time warnings. ( #6501 )
2024-10-21 12:30:49 +02:00
5391663072
Drop Python 3.8 Support ( #6472 )
2024-10-16 10:03:16 +02:00
ceedb026f8
Remove top-level imports that install the reactor from scrapy.extensions.telnet. ( #6432 )
2024-07-09 08:34:58 +02:00
2facdd4fb0
Add change reactor test to CrawlerRunner
2024-05-26 19:55:54 -03:00
fa0c598096
Add component getters to Crawler ( #6181 )
2024-01-15 13:14:02 +01:00
1d81585612
Merge pull request #6112 from wRAR/test-shutdown-forced
...
Make shutdown tests more robust.
2023-10-30 09:35:29 +01:00
6b0c18e921
Improve signal sending in test_shutdown_forced.
2023-10-19 23:13:47 +04:00
Andrey Rakhmatullin
Adrian
Adrián Chaves
Swayam Gupta
Vsevolod Breus
Laerte Pereira