9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint ( #7401 )
...
* chore: simplify some code, get rid of nested fns where it's makes sence
reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation
* chore: address some pylint rules
* remove `pylint: disable=deprecated-class` from `reactor.py`
* fix typo
* fix another typo :)
* revert backward incompatible arg renames
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
8d69a7c865
Deprecate FormRequest in favor of form2request ( #6438 )
...
* WIP
* Add docs
* Remove FormRequest from tests
* Silence mypy issue
* Address docs-tests issues
---------
Co-authored-by: Adrian Chaves <adrian@zyte.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 22:38:29 +05:00
b9c2240040
sphinx-scrapy: 0.8.4 → 0.8.5 ( #7472 )
2026-04-29 14:17:26 +02:00
528745b059
sphinx-scrapy: 0.8.3 → 0.8.4 ( #7467 )
2026-04-28 12:17:12 +02:00
41f43f4649
sphinx-scrapy: 0.7.1 → 0.8.3 ( #7454 )
2026-04-23 10:03:09 +02:00
010faf1722
Use sphinx-scrapy 0.7.1 ( #7406 )
2026-04-06 15:29:59 +02:00
f8d103a65a
Add llms.txt and llms-full.txt generation ( #7380 )
2026-04-06 10:24:21 +02:00
a8e99aeb2e
Fix botocore-pinned and pypy3-* ( #7363 )
2026-03-26 08:21:58 +01:00
6fe27ba33e
Add more no-reactor tests ( #7259 )
...
* Generic changes and scrapy bench.
* scrapy check.
* scrapy crawl.
* scrapy fetch.
* scrapy parse.
* scrapy runspider.
* scrapy shell.
* Skip httpx tests on default-reactor.
* Review requires_reactor marks.
* Make test functions that require an event loop async def.
* Improve test_pending_asyncio_tasks().
* Add Mac OS DNS error.
* Refactor most of test_scheduler.py.
* Finish refactoring DownloaderAwareSchedulerTestMixin.
* Refactor test_engine_loop.py.
* Add the no-reactor-extra-deps tox env, run no-reactor on macos.
* Skip unhandled CancelledError when shutting down the engine.
* Fix typing and pre-commit checks.
* Fix typing problems in master.
---------
Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
e02ad08672
Move to mypy --strict with exceptions. ( #7300 )
...
* Move to mypy --strict with exceptions.
* Fix typing-tests.
2026-03-02 15:47:23 +05:00
ccfa052fa1
Enable in-process HTTP tests without a reactor. ( #7254 )
2026-02-13 19:08:06 +01:00
6e0a0e476a
Fix running tests with -n auto, enable it on CI ( #7257 )
...
* Make test_start_deprecated_super() more robust.
* Work around the FTPFilesStore.FTP_* initialization.
* Enable xdist on CI.
* Add PYTEST_ADDOPTS to tox passenv.
2026-02-13 22:12:57 +05:00
fc30c47f38
Add HttpxDownloader. ( #7239 )
2026-02-12 09:30:44 +01:00
c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. ( #7233 )
...
* Move no-reactor to pytest-asyncio.
* Unify the style.
* Skip a test that installs a reactor in no-reactor.
* Fix pinned env deps.
* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
3186ccf5d5
Reactorless tests 1: the test env ( #6952 )
...
* Reactorless tests.
* Updates after merging master.
* Update a new import.
* Updates after merging master.
* Update testenv:no-reactor-pinned.
* Foundations for the reactorless mode.
* Fixes after merging master.
* Updates after merging reactorless-base.
* Add simple subprocess tests for reactorless AsyncCrawler*.
* More reactorless tests.
* Refactor AsyncCrawlerProcess.start().
* More checks.
* Fix test_reactorless_import_hook.
* More tests.
* Call install_reactor() before asyncio.run().
* Cleanup.
* Rephrase.
* Fix a test.
* Fixes.
* Rephrasing.
* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.
* Fix a new import.
* Clarify --reactor marks.
* Cleanup.
* More docs/comments.
* Update test_fallback_workflow().
* Update TestDeferredFFromCoroF.
* Typing improvements.
2026-02-02 19:25:30 +05:00
813fd9f1ac
Restore brotlicffi support ( #7160 )
...
* restore brotlicffi dependency
* pre-commit
2025-11-25 12:06:55 +05:00
14737e91ed
Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
426aafddca
Don't force the unavailable sysmon coverage core. ( #7137 )
2025-11-13 23:22:52 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses ( #7050 )
2025-09-15 11:24:09 +02:00
6b2997af90
Migrate to pytest-twisted ( #6938 )
...
* Migrate to pytest-twisted (WIP)
* Some typing fixes.
* Make --reactor=asyncio the default again.
* Try installing the correct event loop policy in tests on Windows.
* Make reactor_pytest a normal fixture.
* Fix test warnings.
* Fix FTPDownloadHandler teardown.
* Cleanups, typing.
* More cleanup.
* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
df342eee6e
Bump tool versions. ( #6941 )
2025-07-05 10:30:30 +02:00
91b186cf18
Use new pytest for new Twisted. ( #6893 )
2025-06-23 17:42:11 +02:00
92c18d15b4
Remove ProcessTest and SiteTest. ( #6885 )
...
* Remove ProcessTest and SiteTest.
* Restore the support for Windows line endings in TestParseCommand.
* Add a test for running a scrapy command in a project subdir.
* Remove pywin32 from test deps.
2025-06-13 22:18:27 +05:00
ba10dcfd1a
Pin older Twisted in tests, update type hints. ( #6882 )
2025-06-09 09:53:42 +02:00
036f3e5627
Support asynchronous start requests ( #6729 )
2025-05-07 19:04:03 +02:00
4899d416e7
Add PyPy 3.11 to CI. ( #6697 )
2025-05-06 12:31:28 +02:00
e50914e0f5
Codecov: Add test analytics ( #6741 )
2025-03-21 12:28:47 +01:00
d0dabbc097
Enable AsyncioSelectorReactor by default. ( #6713 )
...
* Enable AsyncioSelectorReactor by default.
* Improve get_crawler(), switch more tests to it.
* Fix the remaining default-reactor test failures.
* Address documentation feedback.
* Make pinned envs more consistent.
2025-03-12 01:18:30 +05:00
803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests ( #6724 )
2025-03-11 14:53:42 +01:00
0c9200094e
Extend BaseSettings with utils for add-ons ( #6614 )
2025-03-05 10:31:59 +01:00
ba5df629a2
Refactor downloader tests ( #6647 )
...
* Make download handler test base classes abstract.
* Small cleanup.
* Don't run the full test suite for special HTTP cases.
* Don't run tests in imported base classes.
* Remove an obsolete service_identity check.
* Move FTP imports back to the top level.
* Simplify the H2DownloadHandler import.
* Forbig pytest 8.2.x.
* Revert "Simplify the H2DownloadHandler import."
This reverts commit ed187046ac
2025-02-03 20:11:47 +05:00
d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. ( #6613 )
2025-01-23 09:22:18 +01:00
3154b08e90
Improve coverage speed on Python 3.12+.
2025-01-07 20:42:03 +05:00
7dfbecd392
Fix tracking of coverage in subprocesses.
2025-01-07 20:41:48 +05:00
f7af7b282d
Bump mypy and stubs.
2024-12-29 16:45:26 +05:00
4a0c05749c
Bump tool versions.
2024-12-29 14:29:27 +05:00
8c23da943c
Integrating configs into pyproject.toml ( #6547 )
2024-11-19 19:21:15 +05:00
d85c39f5bc
Deprecation removals. ( #6500 )
...
* Deprecation removals.
* Clean up the default pytest filterwarnings.
* Remove test_get_images_old().
* Redo boto-requiring test filtering.
* Remove an unused function.
* Improve the Crawler.crawl() error message.
* Fix the test.
2024-10-31 18:06:22 +05:00
d10c58ff38
Bump pyftpdlib to the version supporting Python 3.13 on Windows.
2024-10-22 19:07:21 +05:00
7e07d48cc5
Small 3.7 and 3.8 cleanup.
2024-10-17 23:22:37 +05:00
f65e64a724
Misc typing improvements. ( #6494 )
2024-10-16 18:38:43 +02:00
5391663072
Drop Python 3.8 Support ( #6472 )
2024-10-16 10:03:16 +02:00
87651fdf47
Don't use types-setuptools.
2024-10-08 16:04:26 +05:00
5794071f96
Typing fixes and updates. ( #6460 )
2024-08-23 12:48:01 +02:00
3c9c1a31bc
Bump pylint.
2024-07-11 12:30:12 +05:00
129dbfa0bf
Bump tool versions.
2024-07-11 12:23:43 +05:00
Albert Eduardovich N.
Adrián Chaves
Adrian
Andrey Rakhmatullin
Rui Xi
Suejung Shin
Swayam Gupta
Vsevolod Breus