4b40d2d06a
Close various garbage collectible resources (mostly in tests) ( #7644 )
...
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
2026-06-26 23:05:08 +05:00
edc353c975
Improve test coverage for shell/ ( #7680 )
...
* Improve test coverage for shell/
* Address test issues
* Make the shell config test more reliable on Windows
2026-06-26 19:40:36 +05:00
55c17a8985
Disable mypy allow_any_generics. ( #7532 )
2026-05-18 15:54:06 +02:00
b9be5ce053
Reactorless shell and other small shell improvements ( #7395 )
...
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
2026-04-08 18:24:51 +05:00
b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. ( #7394 )
2026-04-02 10:20:54 +02:00
03d105ac92
Add warnings to components not working without a reactor ( #7355 )
2026-03-25 09:21:08 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
939db88b04
Disable mypy allow_untyped_calls. ( #7331 )
2026-03-24 10:31:17 +01:00
2347138ba4
Unified download handler exceptions ( #7208 )
...
* Add DownloadTimeoutError.
* Don't expect ConnectionAborted in tests.
* Add DownloadCancelledError.
* Add ResponseDataLoss.
* Add more download handler tests.
* Add DownloadConnectionRefusedError.
* Add UnsupportedURLScheme.
* Add CannotResolveHostError.
* Add DownloadFailedError.
* Remove wrapped Twisted exceptions from lists.
* Update a test expectation.
* Extract wrap_twisted_exceptions().
* Wrap TxTimeoutError.
* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
e48a1bdde3
Speed up and refactor command tests ( #7118 )
2025-10-27 12:28:16 +01:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
92c18d15b4
Remove ProcessTest and SiteTest. ( #6885 )
...
* Remove ProcessTest and SiteTest.
* Restore the support for Windows line endings in TestParseCommand.
* Add a test for running a scrapy command in a project subdir.
* Remove pywin32 from test deps.
2025-06-13 22:18:27 +05:00
d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. ( #6873 )
2025-06-06 22:59:09 +02:00
3aa5e75787
Use AsyncCrawlerProcess in commands. ( #6845 )
...
* Use AsyncCrawlerProcess in commands.
* Ignore coverage of abstract methods.
* Address feedback.
2025-06-03 17:47:52 +05:00
5a605969bd
Converting tests to plain asserts, part 2. ( #6699 )
2025-03-06 20:52:41 +01:00
a5731c1944
Move most of the test utils inside tests.
2025-03-02 21:04:12 +05:00
cc9c415bf3
Disable ipython for InteractiveShellTest.
2023-10-18 19:30:57 +04:00
d19e315b0b
Add an interactive test for `scrapy shell`.
2023-09-24 21:51:19 +04:00
b07d3f85a3
Upgrade CI tools
2023-02-02 06:37:40 +01:00
a5c1ef8276
sort imports with isort
2023-01-25 14:25:15 -06:00
c5cdd0d30c
reaplying black again and removing line length on pre-commit black config
2022-12-29 12:48:20 -03:00
303f0a70fc
re applying black to the code with default line length
2022-12-29 11:26:06 -03:00
12b556a352
Merge branch 'master' into add-black-formatter
2022-12-26 12:25:32 -03:00
c2de9372a2
added unittest for scrapy shell command and fixed flake8 error
2022-12-13 21:37:17 -08:00
e211ec0aa2
adding black formatter to all the code
2022-11-29 11:30:46 -03:00
40d9ca3bdd
use pathlib
2022-10-17 17:40:10 -04:00
26c70318cb
make Scrapy testing suite more robust in environments where non-existing hosts are resolvable
2022-07-17 16:47:20 +05:00
58ca8bbf6d
Use f-strings ( #4307 )
2020-08-26 07:11:05 +02:00
6e58da1dcd
Merge branch 'master' into azure-pipelines
2020-07-02 17:49:42 +02:00
3eeecb4273
Merge branch 'master' into flake8-max-line-length
2020-05-15 11:11:20 -03:00
9c6d13075a
Flake8: Remove E128 (tests, part 1)
2020-05-06 18:56:14 -03:00
2f510fd47d
Fix ShellTest.test_local_file on Windows
2020-04-15 21:10:05 +02:00
182394bcec
E501 compliance (tests)
2020-03-22 23:29:30 -03:00
75d6f56c8a
Switch from ` to `` where inline code formatting is desired
2019-03-01 16:56:58 +01:00
53da56c8dc
TST update tests to use get/getall/attrib instead of extract
2018-09-12 17:57:27 +05:00
2cd579a774
Add test for fetch(url) within shell with and without redirect
2016-12-07 19:07:32 +01:00
7e54de2455
Add tests for shell command with and without --no-redirect
2016-12-07 18:41:24 +01:00
b828facff4
Add shell test for using scrapy.Request() directly without importing scrapy
2016-09-15 19:25:20 +02:00
78f00401cd
Remove unused import in tests
2016-01-29 16:56:05 +01:00
481e251775
Move guess_scheme() tests to relevant test module
2016-01-28 13:51:50 +01:00
cae268402d
Move guess_scheme() to scrapy.utils.url
2016-01-28 13:42:04 +01:00
5f09da60c1
Revert "Use pytest.mark.parametrize decorator"
...
This reverts commit 1a30a7774b
2016-01-22 23:48:58 +01:00
1a30a7774b
Use pytest.mark.parametrize decorator
2016-01-22 18:22:19 +01:00
7a51d370f3
Regex-based guess_scheme() + refactor tests
2016-01-22 17:16:27 +01:00
6d73e057b5
Extract guess_scheme function and refactor tests
2016-01-22 13:07:42 +01:00
8bd5b60889
Remove relpath filepath
2016-01-21 23:23:50 +01:00
240ecbf323
Add local file tests for scrapy shell command
...
Continuation of #1579
2016-01-21 22:59:48 +01:00
47d3c63338
py3: port fetch and shell commands, and review + enable already passing test_closespider.py and tests/test_utils_template.py
2016-01-20 23:01:29 +03:00
242c08553f
move tests under root dir
2014-07-30 19:45:08 -03:00
Andrey Rakhmatullin
Adrian
Adrián Chaves
Jalil SA
Emmanuel Rondan
Emmanuel Rondan
Alex
Matthew Donoughe
Mikhail Korobov
Ammar Najjar
Eugenio Lacuesta
Paul Tremberth
Konstantin Lopuhin
Daniel Graña