Commit Graph

28 Commits

Author SHA1 Message Date
Adrian 74e6b61071
Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults (#7665)
* Make DOWNLOADER_CLIENT_TLS_CIPHERS=None enable Twisted defaults

* Remove pointless comment

* Remove unnecessary mypy comments
2026-06-25 16:44:51 +05:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Andrey Rakhmatullin d9e2f5fbf7
Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
Andrey Rakhmatullin f7db039d1c
Update the TLS code to Twisted 26.4.0 (#7347)
* Unpin Twisted, add a twisted-trunk tox env (without http2 for now).

* Hide the _setAcceptableProtocols import.

* Silence ScrapyClientContextFactory HTTP/1.0 warnings.

* Update ScrapyClientTLSOptions for unreleased Twisted.

* Add twisted-trunk to CI.

* Update for new changes in Twisted trunk.

* Xfail a test failing with newer Twisted.

* Test Twisted trunk with extra-deps, update the ALPN code.

* Cleanup.

* Fix typing.

* Update relevant type hints for Twisted trunk.

* Update test_no_context_sharing().

* Silence a weird mypy error.

* Update Twisted versions, unpin pyOpenSSL.

* Update a comment.

* Silence pylint.

* Make a factory fixture.

* Improve the _setAcceptableProtocols comment.

* Set OP_LEGACY_SERVER_CONNECT on new Twisted too.

* Add Twisted[http2] to extra-deps-pinned.
2026-05-12 18:22:31 +05:00
Andrey Rakhmatullin 320e40a044
Revert sharing of the SSL context in _ScrapyClientContextFactory. (#7450) 2026-04-22 19:01:01 +05:00
Albert Eduardovich N. 8835a69f12
Use `dataclass` for the downloader `Slot` (#7405) 2026-04-08 13:26:01 +02:00
Andrey Rakhmatullin 510f09a961
Deprecate ScrapyClientContextFactory. (#7391) 2026-04-02 09:29:47 +02:00
Andrey Rakhmatullin 2ce02d417a
TLS code modernization (#7353)
* Import some stuff from service_identity directly.

* Extract _log_tls().

* Sync validating IP addresses.

* Introduce _deps_compat.

* Deprecate getCertificateOptions().

* Deprecate ScrapyClientTLSOptions.

* Add pragma: no cover.

* verify=False and trustRoot=platformTrust() are the defaults.

* Improve docstrings for TLS classes.

* Deprecate AcceptableProtocolsContextFactory.

* Switch from DefaultOpenSSLContextFactory to CertificateOptions.

* Deprecate load_context_factory_from_settings().

* Remove the outdated warning about incompatible factories.

* Simplify _load_context_factory_from_settings().

* Make CertificateOptions and Context once per factory.

* pragma: no cover
2026-03-25 15:42:43 +05:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin 939db88b04
Disable mypy allow_untyped_calls. (#7331) 2026-03-24 10:31:17 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin c8aa429c9b
Reactorless tests 2: enable pytest-asyncio. (#7233)
* Move no-reactor to pytest-asyncio.

* Unify the style.

* Skip a test that installs a reactor in no-reactor.

* Fix pinned env deps.

* Strict pytest-asyncio mode.
2026-02-02 19:36:23 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin a2463325db
Allow Crawler.crawl_async() without asyncio. (#7202) 2025-12-31 18:33:59 +01:00
Andrey Rakhmatullin baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984)
* Add a crawler argument to MiddlewareManager.__init__().

* Deprecate the spider argument in some MiddlewareManager subclasses.

* Remove the spider argument from ExecutionEngine.open_spider_async().

* Remove the spider argument from Scraper.open_spider().

* Fix the Scraper.close_spider() type hint.

* Remove the spider argument from Downloader internals.

* Deprecate the spider arg of download_func.

* Deprecate the spider arg of Downloader.fetch().

* Deprecate the spider arg of ItemPipelineManager.process_item().

* Fix pylint.

* Tests for MiddlewareManager spider code.

* Deprecated spider arg handling for custom pipeline managers.

* Add a test for the deprecated spider arg in a custom downloader.

* Remove TODO.

* Fix pylint.
2025-08-05 12:21:08 +05:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960)
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938)
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 0d2d2892ba Silence the readBody warning. 2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin bc1aeeefc9 Deprecate overriding ScrapyClientContextFactory.getContext(). 2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Adrián Chaves f57fc454be Replace Slot.throttle with Request.meta['dont_throttle'] 2024-11-05 19:44:30 +01:00
Adrián Chaves d7581c6b41
Allow disabling the AutoThrottle extension for a given slot (#6246) 2024-03-12 09:44:29 +01:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Ammar Najjar 0e57918231
test(Slot): cover __repr__
Issue: #4324
2020-08-28 13:58:32 +02:00