Commit Graph

12 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 4e956bd2de
Add support for HTTP/2 and for SOCKS proxies to HttpxDownloadHandler, improve handler docs (#7575)
* Add support for HTTP/2 and SOCKS proxies to HttpxDownloadHandler.

* Update the docs.

* Trim the tables.

* Restore lost wording.

* Handlers docs improvements and fixes.
2026-06-09 01:10:44 +05:00
Andrey Rakhmatullin d9e2f5fbf7
Add settings for TLS min/max version as a replacement for the TLS method (#6546) 2026-06-08 11:54:10 +02:00
Andrey Rakhmatullin f875af4a86
Add BaseStreamingDownloadHandler and migrate HttpxDownloadHandler to it. (#7524) 2026-05-18 15:52:13 +02:00
Andrey Rakhmatullin 85c616c5c7
Fix certificate issuer verification on new Twisted. (#7520) 2026-05-14 19:56:05 +02:00
Andrey Rakhmatullin 7fc84d372a
Cleanup/clarifications of proxy support and improvements for proxy tests (#7496)
* Run mitmproxy-based tests for every handler and improve them.

* Fixes for H2DownloadHandler.

* Revert "Fixes for H2DownloadHandler."

This reverts commit bcdbd097cd.

* Add TestHttpsProxy for HTTP11DownloadHandler.

* Use the configured context factory in ScrapyProxyAgent.

* Reword.

* Reword.

* pragma: no cover for H2DownloadHandler proxy code.

* Raise an exception for HTTPS proxies instead of using HTTP.

* Remove non-working proxy support and explicitly forbid HTTP.

* pragma: no cover

* Add a docs note about HTTPS proxies.

* Rename ScrapyProxyAgent.
2026-05-12 17:45:54 +05:00
Andrey Rakhmatullin c62a81d7dd
Drop HTTP/1.0 code. (#7486) 2026-05-04 08:50:41 +02:00
Andrey Rakhmatullin 27092b2cb7
Small download handler fixes and improvements (#7441)
* Popen cleanup.

* Improve the test for response headers.

* Fix default headers of Httpx and H2 handlers, add tests.

* Check Location in redirect tests.

* Simplify bindaddress tests.

* Add download_latency to HttpxDownloadHandler, add tests.

* Remove explicit method="GET".

* Don't expect a specific value in test_download_with_maxsize_very_large_file

* Use a non-existent scheme in test_unsupported_scheme.

* Remporarily rollback the test_download_latency test check.

* Better check in test_download_latency.

* Improve logic for StreamCloseReason.CANCELLED.
2026-04-21 18:27:16 +05:00
Andrey Rakhmatullin eabb149f4b
Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY, add DOWNLOAD_VERIFY_CERTIFICATES (#7379)
* Deprecate DOWNLOADER_CLIENTCONTEXTFACTORY.

* Deprecate BrowserLikeContextFactory.

* Add DOWNLOAD_VERIFY_CERTIFICATES.

* Cleanup, add acceptableCiphers to the verifying path.
2026-04-01 15:18:43 +05:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin 9f4651151d
Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. (#7276) 2026-03-24 10:36:48 +01:00
Julian Hamze c148ec4433
Add a DOWNLOAD_BIND_ADDRESS setting for download handlers (#7283) 2026-03-19 09:16:40 +01:00
Andrey Rakhmatullin fc30c47f38
Add HttpxDownloader. (#7239) 2026-02-12 09:30:44 +01:00