609f64c55d
Make brotli a hard dependency ( #7929 )
2026-08-09 19:24:14 +02:00
8e94df6526
Time out hanging tests in the VCS dependencies job ( #7960 )
...
* Time out hanging tests in the VCS dependencies job
* Do not test pydispatcher’s latest commit
2026-08-09 21:18:33 +05:00
94dff69468
Add VCS CI job, fix support for upcoming parsel version ( #7924 )
2026-08-09 14:15:58 +05:00
0c542afa65
Upgrade the minimum queuelib to 1.6.1 ( #7874 )
2026-08-04 18:47:00 +05:00
a7385d6e51
Type tests.mockserver and other resource-defining files ( #7865 )
...
* Type tests.mockserver and other resource-defining files
* Address pylint issues
2026-08-03 23:44:28 +05:00
24de06bcd3
CI: Install dependencies with uv ( #7838 )
...
* CI: Install dependencies with uv
* Setup GitHub Actions hardening
* CI: Use uv for mitmproxy, benchmarks and cache keys
* Fix mitmproxy install on PyPy
2026-07-31 23:02:52 +05:00
8caaac6ecb
fix(shell): Run IPython prompt in a thread under a running loop ( #7816 )
2026-07-31 18:05:22 +02:00
6cefaa5434
Add a stats reference ( #7814 )
2026-07-31 15:43:40 +05:00
aa5ded2539
Set up CodSpeed ( #7831 )
...
* Set up CodSpeed
* CodSpeed: update permissions
2026-07-31 13:27:14 +05:00
7e8b58a2b2
Migrate away from testfixtures. ( #7793 )
2026-07-28 09:52:52 +02:00
56dee203e9
Use httpx2 as a preferred alternative to httpx. ( #7762 )
2026-07-23 10:29:22 +02:00
a54c438da1
Provide extras for optional dependencies ( #7596 )
2026-07-22 10:44:26 +02:00
b1b9efb473
Rewrite GCSFilesStore tests to use mocking. ( #7727 )
2026-07-21 11:43:36 +02:00
c9446931a8
Use external mitmproxy. ( #7720 )
2026-07-09 17:02:05 +02:00
c5ec881f1d
Update tools ( #7724 )
2026-07-07 17:27:26 +05:00
4b40d2d06a
Close various garbage collectible resources (mostly in tests) ( #7644 )
...
* Set TELNETCONSOLE_ENABLED=False in get_crawler().
* Make _get_console_and_portal() a context manager.
* Allow PYTHONTRACEMALLOC in tox.
* Close the event loop in test_custom_asyncio_loop_enabled_false().
* Close the handler in _uninstall_scrapy_root_handler().
* Close queues in test_squeues.py.
* Close empty queues in init_prios().
* Close PopenSpawn stdin and stdout explicitly.
* Silence the unclosed socket warning.
* Implement more methods in CustomStatsCollector.
* Link the Twisted issue.
* Close the connection on download_maxsize.
* Fix typing.
* Add a comment about CustomStatsCollector.
* Restore the coverage.
* Properly close fixture queues.
* More robust file closing in feed storages.
* Close the file in TestMarshalItemExporter.test_nonstring_types_item().
* Cleanup temporary file handling in test_exporters.py.
* Close the file in test_stats_file_failed().
* Use an explicit TextIOWrapper in XmlItemExporter.
2026-06-26 23:05:08 +05:00
f605defefc
Document scrapy-lint, remove start_url check ( #7627 )
...
* Document scrapy-lint, remove start_url check
* Remove the offsite URL check in favor of scrapy-lint
2026-06-22 20:12:16 +05:00
9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. ( #7619 )
2026-06-15 08:44:33 +02:00
2d0a898e2d
tox.ini: pinned, typing → min, mypy ( #7595 )
2026-06-11 12:10:48 +05:00
ddafb37a7c
Assorted test fixes ( #7585 )
...
* Fix the mitmproxy junitxml file name.
* Fix TestAsyncCrawlerRunnerHasSpider regression.
* Update split out file refs.
* Update the test_counter_handler() docstring.
2026-06-10 11:01:06 +02:00
4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs ( #7575 )
...
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.
* Update the docs.
* Trim the tables.
* Restore lost wording.
* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
b2d8b06be6
Support sending requests with "unsafe" URLs ( #7473 )
2026-06-05 11:51:03 +02:00
55c17a8985
Disable mypy allow_any_generics. ( #7532 )
2026-05-18 15:54:06 +02:00
f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. ( #7524 )
2026-05-18 15:52:13 +02:00
f7db039d1c
Update the TLS code to Twisted 26.4.0 ( #7347 )
...
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).
* Hide the _setAcceptableProtocols import.
* Silence ScrapyClientContextFactory HTTP/1.0 warnings.
* Update ScrapyClientTLSOptions for unreleased Twisted.
* Add twisted-trunk to CI.
* Update for new changes in Twisted trunk.
* Xfail a test failing with newer Twisted.
* Test Twisted trunk with extra-deps, update the ALPN code.
* Cleanup.
* Fix typing.
* Update relevant type hints for Twisted trunk.
* Update test_no_context_sharing().
* Silence a weird mypy error.
* Update Twisted versions, unpin pyOpenSSL.
* Update a comment.
* Silence pylint.
* Make a factory fixture.
* Improve the _setAcceptableProtocols comment.
* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.
* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
7f15ca92fc
sphinx-scrapy: 0.8.5 → 0.8.6 ( #7507 )
2026-05-12 12:49:23 +05:00
fc14a0ce59
Enable mypy warn_return_any. ( #7492 )
2026-05-04 22:24:47 +02:00
9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint ( #7401 )
...
* chore: simplify some code, get rid of nested fns where it's makes sence
reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation
* chore: address some pylint rules
* remove `pylint: disable=deprecated-class` from `reactor.py`
* fix typo
* fix another typo :)
* revert backward incompatible arg renames
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
8d69a7c865
Deprecate FormRequest in favor of form2request ( #6438 )
...
* WIP
* Add docs
* Remove FormRequest from tests
* Silence mypy issue
* Address docs-tests issues
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 22:38:29 +05:00
b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 ( #7472 )
2026-04-29 14:17:26 +02:00
528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 ( #7467 )
2026-04-28 12:17:12 +02:00
41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 ( #7454 )
2026-04-23 10:03:09 +02:00
010faf1722
Use sphinx-scrapy 0.7.1 ( #7406 )
2026-04-06 15:29:59 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
a8e99aeb2e
Fix botocore-pinned and pypy3-* ( #7363 )
2026-03-26 08:21:58 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
ccfa052fa1
Enable in-process HTTP tests without a reactor. ( #7254 )
2026-02-13 19:08:06 +01:00
6e0a0e476a
Fix running tests with -n auto, enable it on CI ( #7257 )
...
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
813fd9f1ac
Restore brotlicffi support ( #7160 )
...
* restore brotlicffi dependency
* pre-commit
2025-11-25 12:06:55 +05:00
14737e91ed
Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
426aafddca
Don't force the unavailable sysmon coverage core. ( #7137 )
2025-11-13 23:22:52 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses ( #7050 )
2025-09-15 11:24:09 +02:00
Adrian
Laerte Pereira
Andrey Rakhmatullin
Albert Eduardovich N.
Adrián Chaves
Rui Xi