c9f952c258
Refactor and improve catching warnings in tests. ( #7643 )
2026-06-19 21:04:34 +02:00
55c17a8985
Disable mypy allow_any_generics. ( #7532 )
2026-05-18 15:54:06 +02:00
068aa69b35
Fixes for Python 3.14. ( #7460 )
2026-04-27 18:47:52 +05:00
9da14cdff1
Fix file extension extraction for URLs with query parameters ( #4225 ) ( #7414 )
...
* Fix file extension extraction for URLs with query parameters (#4225 )
* Prioritize parsed path for file extensions and add tests (#4225 )
2026-04-14 20:04:38 +05:00
bfe34492fa
asyncio alternative to deferToThread() ( #7349 )
2026-03-24 15:41:15 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
d42b23d78a
Await persist_file() in media pipelines. ( #7182 )
2026-03-24 10:52:56 +01:00
7010985e4f
Bump ruff, fix some of rules ( #7277 )
2026-02-23 11:48:38 +01:00
6e0a0e476a
Fix running tests with -n auto, enable it on CI ( #7257 )
...
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
09bd8f4231
Properly close ftplib.FTP(). ( #7256 )
2026-02-13 19:12:13 +05:00
11977afba5
Rename async test decorators. ( #7234 )
2026-02-02 20:39:43 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
c097921c44
Extract @_warn_spider_arg. ( #7033 )
...
* Extract @_warn_spider_arg.
* Also deprecate passing spider=None.
2025-09-01 20:17:13 +05:00
3ee4a52aa1
Raise NotConfigured when FILES_STORE or IMAGES_STORE is None or empty ( #6969 )
2025-09-01 11:39:39 +02:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
8c8f4ff033
Extract in-process test HTTP servers into separate processes ( #6960 )
...
* Switch TestEngineBase to mockserver.
* Split mockserver.py.
* Stop using WrappingFactory.
* Remove keyfile/certfile fields from TestHttpBase.
* Move webclient-only resources to test_webclient.py.
* Refactor MockServer.url().
* Migrate TestHttpBase to mockserver.
* Migrate TestSimpleHttpsBase to a separate mockserver.
* Rename TestHttpMockServerBase.
* Migrate TestHttpProxyBase to a separate mockserver.
* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
9e83a58643
Add type check in get_media_requests to enforce image / file urls as a list ( #6949 )
2025-07-19 11:54:35 +02:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
b4d11b8b25
Further reduce deps on unittest. ( #6884 )
2025-06-11 01:28:09 +02:00
d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. ( #6873 )
2025-06-06 22:59:09 +02:00
eb654aa1a8
Convert remaining unittest assert* calls, use the tmp_path fixture. ( #6725 )
2025-03-11 18:00:36 +01:00
02ed71d887
Converting tests to plain asserts, part 6. ( #6709 )
2025-03-09 20:20:24 +01:00
a5731c1944
Move most of the test utils inside tests.
2025-03-02 21:04:12 +05:00
cec0aeca58
Bump ruff, switch from black to ruff-format ( #6631 )
2025-01-27 11:07:09 +01:00
c87354cd46
Enable SIM Ruff rules.
2025-01-02 01:03:36 +05:00
93644f2c30
Add flake8-pie rules to ruff.
2024-12-12 20:33:35 +05:00
cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules ( #6577 )
2024-12-10 19:53:27 +01:00
ab5cb7c7d9
Merge pull request #6540 from wRAR/build_from_settings
...
Remove build_from_settings() and deprecate from_settings() code
2024-11-14 20:40:54 +05:00
6aa4d2b4ab
Prefer crawler.settings over settings in media pipelines.
2024-11-14 02:03:43 +05:00
28fafbb8c5
Modernize the media pipeline initialization API.
2024-11-14 01:29:51 +05:00
f796d8780c
Add tests for MediaPipeline.from_crawler() and related code.
2024-11-12 21:08:04 +05:00
b042ad255d
Address some previously ignored pylint messages. ( #6531 )
2024-11-11 11:49:52 +01:00
d85c39f5bc
Deprecation removals. ( #6500 )
...
* Deprecation removals.
* Clean up the default pytest filterwarnings.
* Remove test_get_images_old().
* Redo boto-requiring test filtering.
* Remove an unused function.
* Improve the Crawler.crawl() error message.
* Fix the test.
2024-10-31 18:06:22 +05:00
5391663072
Drop Python 3.8 Support ( #6472 )
2024-10-16 10:03:16 +02:00
26a16f2c43
Fix tests
2024-02-28 16:36:19 -03:00
b4acf5c827
Fix and remove most of the entries from the mypy ignore list ( #6137 )
2023-11-07 09:34:35 +01:00
1ed9ed4f92
Require pyftpdlib that supports Python 3.12.
2023-10-02 21:33:02 +04:00
c4f0aa4fdf
Refactor test cases to improve unit test quality ( #5986 )
2023-07-26 13:09:03 +02:00
e058a05763
Skip tests that don't work on Python 3.12.
2023-07-22 17:51:13 +04:00
712ee98848
Merge branch 'master' into typing-utils
2023-06-24 18:31:56 +04:00
85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export
...
Fix #960 : S3 Feed Export throws boto error
2023-06-13 18:44:06 +04:00
54fa04aa0a
Add typing to scrapy/utils/test.py, fix a FTP test.
2023-05-07 22:39:27 +04:00
d47c732ae9
base64-decode GCS checksums ( #5891 )
2023-04-11 18:25:42 +02:00
39dbfa1d82
Minimize test reliance on S3; do not install botocore on the default test environments
2023-03-15 15:31:39 +01:00
4692e0e16b
Merge branch 'master' into path_object_error_#5739
2023-02-02 11:10:01 -08:00
Andrey Rakhmatullin
Hamzah Alshawwaf
M Ikram Ullah Khan
Vsevolod Breus
Laerte Pereira
freddiewanah
Kartik Kumar
Adrián Chaves
Alex