Commit Graph

319 Commits

Author SHA1 Message Date
aliowka b905a6b2a3 Revert PyPy envs back to plain pytest
PyPy environments explicitly disable coverage to avoid the significant
runtime overhead it adds. Without coverage data, delta has no accurate
mapping for these envs and falls back to the CPython mapping, causing:
- Unreliable test selection (CPython-mapped tests may fail on PyPy)
- Pre-existing PyPy/min-deps failures surfacing unexpectedly (exit code 1)

Affected envs: pypy3, pypy3-extra-deps, min-pypy3

This follows the same principle as the botocore/mitmproxy fix:
delta is only used where it has accurate coverage mapping.
2026-06-17 10:40:03 +03:00
aliowka 699cdb93ed Fix exit code 5 in botocore/mitmproxy tox envs
These envs filter tests with -m requires_botocore / -m requires_mitmproxy.
When delta selects non-matching tests based on the diff, pytest finds no
tests to run for that marker and exits with code 5 (no tests collected).

Revert these three envs back to plain pytest since they are already
highly filtered and delta provides no benefit here.
2026-06-17 10:36:51 +03:00
aliowka bfc753dbee Integrate Delta into all tox environments and clean up workflow
- Replace all bare pytest invocations with delta run -- in tox.ini:
  [min], [testenv:pypy3], [testenv:min-pypy3], [testenv:botocore],
  [testenv:min-botocore], [testenv:mitmproxy]
- Add deltatest-cli to [min] and [testenv:min-pypy3] deps
- Add passenv DELTA_* GITHUB_* to [min] and [testenv:min-pypy3]
- Add Configure Delta Cloud step to tests job in tests-ubuntu.yml
  (writes ~/.delta/config.toml from secrets before tox runs)
- Remove standalone delta pre-check job; delta now runs inline
  in every matrix runner via tox
2026-06-17 10:29:07 +03:00
aliowka 5150b80785 Integrate DeltaTest into tox configuration 2026-06-17 03:13:22 +03:00
Adrian 2d0a898e2d
tox.ini: pinned, typing → min, mypy (#7595) 2026-06-11 12:10:48 +05:00
Andrey Rakhmatullin ddafb37a7c
Assorted test fixes (#7585)
* Fix the mitmproxy junitxml file name.

* Fix TestAsyncCrawlerRunnerHasSpider regression.

* Update split out file refs.

* Update the test_counter_handler() docstring.
2026-06-10 11:01:06 +02:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575)
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
Adrian b2d8b06be6
Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
Andrey Rakhmatullin 55c17a8985
Disable mypy allow_any_generics. (#7532) 2026-05-18 15:54:06 +02:00
Andrey Rakhmatullin f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. (#7524) 2026-05-18 15:52:13 +02:00
Andrey Rakhmatullin f7db039d1c
Update the TLS code to Twisted 26.4.0 (#7347)
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).

* Hide the _setAcceptableProtocols import.

* Silence ScrapyClientContextFactory HTTP/1.0 warnings.

* Update ScrapyClientTLSOptions for unreleased Twisted.

* Add twisted-trunk to CI.

* Update for new changes in Twisted trunk.

* Xfail a test failing with newer Twisted.

* Test Twisted trunk with extra-deps, update the ALPN code.

* Cleanup.

* Fix typing.

* Update relevant type hints for Twisted trunk.

* Update test_no_context_sharing().

* Silence a weird mypy error.

* Update Twisted versions, unpin pyOpenSSL.

* Update a comment.

* Silence pylint.

* Make a factory fixture.

* Improve the _setAcceptableProtocols comment.

* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.

* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
Adrian 7f15ca92fc
sphinx-scrapy: 0.8.5 → 0.8.6 (#7507) 2026-05-12 12:49:23 +05:00
Andrey Rakhmatullin fc14a0ce59
Enable mypy warn_return_any. (#7492) 2026-05-04 22:24:47 +02:00
Albert Eduardovich N. 9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401)
* chore: simplify some code, get rid of nested fns where it's makes sence

reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation

* chore: address some pylint rules

* remove `pylint: disable=deprecated-class` from `reactor.py`

* fix typo

* fix another typo :)

* revert backward incompatible arg renames

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
Adrián Chaves 8d69a7c865
Deprecate FormRequest in favor of form2request (#6438)
* WIP

* Add docs

* Remove FormRequest from tests

* Silence mypy issue

* Address docs-tests issues

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 22:38:29 +05:00
Adrian b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 (#7472) 2026-04-29 14:17:26 +02:00
Adrian 528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 (#7467) 2026-04-28 12:17:12 +02:00
Adrian 41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 (#7454) 2026-04-23 10:03:09 +02:00
Adrian 010faf1722
Use sphinx-scrapy 0.7.1 (#7406) 2026-04-06 15:29:59 +02:00
Adrian f8d103a65a
Add llms.txt and llms-full.txt generation (#7380) 2026-04-06 10:24:21 +02:00
Andrey Rakhmatullin a8e99aeb2e
Fix botocore-pinned and pypy3-* (#7363) 2026-03-26 08:21:58 +01:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin e02ad08672
Move to mypy --strict with exceptions. (#7300)
* Move to mypy --strict with exceptions.

* Fix typing-tests.
2026-03-02 15:47:23 +05:00
Andrey Rakhmatullin ccfa052fa1
Enable in-process HTTP tests without a reactor. (#7254) 2026-02-13 19:08:06 +01:00
Andrey Rakhmatullin 6e0a0e476a
Fix running tests with -n auto, enable it on CI (#7257)
* Make test_start_deprecated_super() more robust.

* Work around the FTPFilesStore.FTP_* initialization.

* Enable xdist on CI.

* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Rui Xi 813fd9f1ac
Restore brotlicffi support (#7160)
* restore brotlicffi dependency

* pre-commit
2025-11-25 12:06:55 +05:00
Rui Xi 14737e91ed
Mitigate brotli and deflate decompression bombs DoS (#7134)
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
Andrey Rakhmatullin 426aafddca
Don't force the unavailable sysmon coverage core. (#7137) 2025-11-13 23:22:52 +05:00
Andrey Rakhmatullin 804ae167df
Update tool versions (#7127) 2025-10-27 14:11:31 +01:00
Andrey Rakhmatullin d414d393d4
Remove Python 3.9 support (#7121) 2025-10-27 12:37:49 +01:00
Adrián Chaves eb496470f1
Add missing doc deps for new examples (#7116) 2025-10-24 11:58:18 +02:00
Andrey Rakhmatullin c577771838
Drop tests/upper-constraints.txt. (#7073) 2025-10-06 11:32:20 +02:00
Andrey Rakhmatullin 393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses (#7050) 2025-09-15 11:24:09 +02:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938)
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 91b186cf18
Use new pytest for new Twisted. (#6893) 2025-06-23 17:42:11 +02:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885)
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 4899d416e7
Add PyPy 3.11 to CI. (#6697) 2025-05-06 12:31:28 +02:00
Suejung Shin e50914e0f5
Codecov: Add test analytics (#6741) 2025-03-21 12:28:47 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713)
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647)
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Andrey Rakhmatullin 3154b08e90 Improve coverage speed on Python 3.12+. 2025-01-07 20:42:03 +05:00