f605defefc
Document scrapy-lint, remove start_url check ( #7627 )
...
* Document scrapy-lint, remove start_url check
* Remove the offsite URL check in favor of scrapy-lint
2026-06-22 20:12:16 +05:00
9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. ( #7619 )
2026-06-15 08:44:33 +02:00
2d0a898e2d
tox.ini: pinned, typing → min, mypy ( #7595 )
2026-06-11 12:10:48 +05:00
ddafb37a7c
Assorted test fixes ( #7585 )
...
* Fix the mitmproxy junitxml file name.
* Fix TestAsyncCrawlerRunnerHasSpider regression.
* Update split out file refs.
* Update the test_counter_handler() docstring.
2026-06-10 11:01:06 +02:00
4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs ( #7575 )
...
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
b2d8b06be6
Support sending requests with "unsafe" URLs ( #7473 )
2026-06-05 11:51:03 +02:00
55c17a8985
Disable mypy allow_any_generics. ( #7532 )
2026-05-18 15:54:06 +02:00
f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. ( #7524 )
2026-05-18 15:52:13 +02:00
f7db039d1c
Update the TLS code to Twisted 26.4.0 ( #7347 )
...
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).
* Hide the _setAcceptableProtocols import.
* Silence ScrapyClientContextFactory HTTP/1.0 warnings.
* Update ScrapyClientTLSOptions for unreleased Twisted.
* Add twisted-trunk to CI.
* Update for new changes in Twisted trunk.
* Xfail a test failing with newer Twisted.
* Test Twisted trunk with extra-deps, update the ALPN code.
* Cleanup.
* Fix typing.
* Update relevant type hints for Twisted trunk.
* Update test_no_context_sharing().
* Silence a weird mypy error.
* Update Twisted versions, unpin pyOpenSSL.
* Update a comment.
* Silence pylint.
* Make a factory fixture.
* Improve the _setAcceptableProtocols comment.
* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.
* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
7f15ca92fc
sphinx-scrapy: 0.8.5 → 0.8.6 ( #7507 )
2026-05-12 12:49:23 +05:00
fc14a0ce59
Enable mypy warn_return_any. ( #7492 )
2026-05-04 22:24:47 +02:00
9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint ( #7401 )
...
* chore: simplify some code, get rid of nested fns where it's makes sence
reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation
* chore: address some pylint rules
* remove `pylint: disable=deprecated-class` from `reactor.py`
* fix typo
* fix another typo :)
* revert backward incompatible arg renames
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
8d69a7c865
Deprecate FormRequest in favor of form2request ( #6438 )
...
* WIP
* Add docs
* Remove FormRequest from tests
* Silence mypy issue
* Address docs-tests issues
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 22:38:29 +05:00
b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 ( #7472 )
2026-04-29 14:17:26 +02:00
528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 ( #7467 )
2026-04-28 12:17:12 +02:00
41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 ( #7454 )
2026-04-23 10:03:09 +02:00
010faf1722
Use sphinx-scrapy 0.7.1 ( #7406 )
2026-04-06 15:29:59 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
a8e99aeb2e
Fix botocore-pinned and pypy3-* ( #7363 )
2026-03-26 08:21:58 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
ccfa052fa1
Enable in-process HTTP tests without a reactor. ( #7254 )
2026-02-13 19:08:06 +01:00
6e0a0e476a
Fix running tests with -n auto, enable it on CI ( #7257 )
...
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
813fd9f1ac
Restore brotlicffi support ( #7160 )
...
* restore brotlicffi dependency
* pre-commit
2025-11-25 12:06:55 +05:00
14737e91ed
Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
426aafddca
Don't force the unavailable sysmon coverage core. ( #7137 )
2025-11-13 23:22:52 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses ( #7050 )
2025-09-15 11:24:09 +02:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
91b186cf18
Use new pytest for new Twisted. ( #6893 )
2025-06-23 17:42:11 +02:00
92c18d15b4
Remove ProcessTest and SiteTest. ( #6885 )
...
* Remove ProcessTest and SiteTest.
* Restore the support for Windows line endings in TestParseCommand.
* Add a test for running a scrapy command in a project subdir.
* Remove pywin32 from test deps.
2025-06-13 22:18:27 +05:00
ba10dcfd1a
Pin older Twisted in tests, update type hints. ( #6882 )
2025-06-09 09:53:42 +02:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
4899d416e7
Add PyPy 3.11 to CI. ( #6697 )
2025-05-06 12:31:28 +02:00
e50914e0f5
Codecov: Add test analytics ( #6741 )
2025-03-21 12:28:47 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests ( #6724 )
2025-03-11 14:53:42 +01:00
0c9200094e
Extend BaseSettings with utils for add-ons ( #6614 )
2025-03-05 10:31:59 +01:00
ba5df629a2
Refactor downloader tests ( #6647 )
...
* Make download handler test base classes abstract.
* Small cleanup.
* Don't run the full test suite for special HTTP cases.
* Don't run tests in imported base classes.
* Remove an obsolete service_identity check.
* Move FTP imports back to the top level.
* Simplify the H2DownloadHandler import.
* Forbig pytest 8.2.x.
* Revert "Simplify the H2DownloadHandler import."
This reverts commit ed187046ac
2025-02-03 20:11:47 +05:00
d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. ( #6613 )
2025-01-23 09:22:18 +01:00
3154b08e90
Improve coverage speed on Python 3.12+.
2025-01-07 20:42:03 +05:00
7dfbecd392
Fix tracking of coverage in subprocesses.
2025-01-07 20:41:48 +05:00
f7af7b282d
Bump mypy and stubs.
2024-12-29 16:45:26 +05:00
Adrian
Andrey Rakhmatullin
Albert Eduardovich N.
Adrián Chaves
Rui Xi
Suejung Shin