Commit Graph

34 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 3f3cb885ed
Address warnings in tests. (#7637) 2026-06-19 06:55:02 +02:00
Andrey Rakhmatullin fada8be1db
Fix Proxy-Authorization handling in BaseStreamingDownloadHandler (#7630) 2026-06-17 11:56:22 +02:00
Andrey Rakhmatullin 4f241b73be
Fix deprecation warnings with pytest 9.1. (#7621) 2026-06-15 08:45:26 +02:00
Andrey Rakhmatullin 9893a7fac6
Fix deprecation warnings with pyOpenSSL 26.3.0. (#7619) 2026-06-15 08:44:33 +02:00
Andrey Rakhmatullin 3a36955261
Assorted test fixes (#7616) 2026-06-15 08:39:58 +02:00
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575)
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
Andrey Rakhmatullin d9e2f5fbf7
Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
Adrian b2d8b06be6
Support sending requests with "unsafe" URLs (#7473) 2026-06-05 11:51:03 +02:00
Andrey Rakhmatullin f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. (#7524) 2026-05-18 15:52:13 +02:00
Andrey Rakhmatullin 85c616c5c7
Fix certificate issuer verification on new Twisted. (#7520) 2026-05-14 19:56:05 +02:00
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496)
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
2026-05-12 17:45:54 +05:00
Andrey Rakhmatullin c62a81d7dd
Drop HTTP/1.0 code. (#7486) 2026-05-04 08:50:41 +02:00
Andrey Rakhmatullin 988afe1454
Clarify dataloss/http2 tests. (#7466) 2026-04-28 14:35:41 +02:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441)
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
2026-04-21 18:27:16 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00
Andrey Rakhmatullin 2e53d90e4c
Refactor HTTP download handlers (#7238) 2026-02-03 20:13:30 +01:00
Andrey Rakhmatullin 11977afba5
Rename async test decorators. (#7234) 2026-02-02 20:39:43 +01:00
Andrey Rakhmatullin 3186ccf5d5
Reactorless tests 1: the test env (#6952)
* Reactorless tests.

* Updates after merging master.

* Update a new import.

* Updates after merging master.

* Update testenv:no-reactor-pinned.

* Foundations for the reactorless mode.

* Fixes after merging master.

* Updates after merging reactorless-base.

* Add simple subprocess tests for reactorless AsyncCrawler*.

* More reactorless tests.

* Refactor AsyncCrawlerProcess.start().

* More checks.

* Fix test_reactorless_import_hook.

* More tests.

* Call install_reactor() before asyncio.run().

* Cleanup.

* Rephrase.

* Fix a test.

* Fixes.

* Rephrasing.

* Set TELNETCONSOLE_ENABLED=False in the reactorless mode.

* Fix a new import.

* Clarify --reactor marks.

* Cleanup.

* More docs/comments.

* Update test_fallback_workflow().

* Update TestDeferredFFromCoroF.

* Typing improvements.
2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin 54d8562fb0
Assorted test improvements (#7232)
* Await some missed deferreds.

* Simplify test_engine_stop_download_*.

* Fix the header value.

* Extend TestHttpWithCrawlerBase.

* Make test_tls_logging more universal.

* Add more tests for header handling.

* Clarify certificate and ip_address integration tests.

* Make test_download_with_maxsize_very_large_file() more universal.

* Fix test_coroutine_asyncio().

* Typo.
2026-02-02 09:37:39 +01:00
Andrey Rakhmatullin 3ec6ae05c1
Fix omitting repeated dataloss warnings in HTTP11DownloadHandler. (#7222) 2026-01-22 00:41:27 +05:00
Andrey Rakhmatullin 2347138ba4
Unified download handler exceptions (#7208)
* Add DownloadTimeoutError.

* Don't expect ConnectionAborted in tests.

* Add DownloadCancelledError.

* Add ResponseDataLoss.

* Add more download handler tests.

* Add DownloadConnectionRefusedError.

* Add UnsupportedURLScheme.

* Add CannotResolveHostError.

* Add DownloadFailedError.

* Remove wrapped Twisted exceptions from lists.

* Update a test expectation.

* Extract wrap_twisted_exceptions().

* Wrap TxTimeoutError.

* Rename ResponseDataLoss and UnsupportedURLScheme.
2026-01-21 23:09:54 +05:00
Andrey Rakhmatullin d1bd8eb49f
Async API for download handlers. (#7164) 2025-12-31 19:03:16 +01:00
Laerte Pereira 11073c8680
Deprecate spider attributes that can be replaced by settings, round 2 (#7039)
* Move duplicate code to utils

* move new function to end

* draft

* update docs

* Update tests

* Update test name

* rename test

* rollback signature

* leftover

* sort

* Rollback some changes

* Rollback download_delay warning

* Rollback

* fix checks

* Remove unused imports

* Add pragma: no cover.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-01 01:26:17 +05:00
Wes Galbraith e98c1644ce
Add unit tests that check download handlers behave like basic http clients (#6946) (#7059) 2025-10-20 13:08:26 +02:00
Andrey Rakhmatullin 552f2fb91e
Generic improvements related to reactorless tests (#6968)
* Skip doctests that import the reactor.

* Extract FTPDownloadHandler tests.

* Skip TestHttps2ClientProtocol via pytestmark.

* Replace some explicit reactor.callLater() in tests.

* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
Andrey Rakhmatullin 8c8f4ff033
Extract in-process test HTTP servers into separate processes (#6960)
* Switch TestEngineBase to mockserver.

* Split mockserver.py.

* Stop using WrappingFactory.

* Remove keyfile/certfile fields from TestHttpBase.

* Move webclient-only resources to test_webclient.py.

* Refactor MockServer.url().

* Migrate TestHttpBase to mockserver.

* Migrate TestSimpleHttpsBase to a separate mockserver.

* Rename TestHttpMockServerBase.

* Migrate TestHttpProxyBase to a separate mockserver.

* Refactor HTTP mockservers.
2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938)
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin 8b3c3ea4ae
Refactor test functions that were returning a Deferred directly. (#6889) 2025-06-30 09:19:23 +02:00
Andrey Rakhmatullin d825133284
Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873) 2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin 8ae418df44
Rewrite download handler tests to coroutines. (#6846) 2025-06-02 16:02:08 +02:00
Andrey Rakhmatullin 916fe50974
Fix and ban the top-level twisted.internet.reactor imports. (#6835) 2025-05-28 15:53:52 +05:00
Andrey Rakhmatullin b41aea4873
Restructure download handler tests. (#6821)
* Restructure download handler tests.

* Typo.

* Use mixins to reduce boilerplate.
2025-05-27 18:19:47 +05:00