diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 7e672f0b4..54a7f0e7c 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -206,6 +206,57 @@ uses the HTTP/1.1 protocol for them. It's implemented using :mod:`twisted.web.client`. +HttpxDownloadHandler +-------------------- + +.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler + +| Supported schemes: ``http``, ``https``. +| Lazy: no. + +This handler supports ``http://host/path`` and ``https://host/path`` URLs and +uses the HTTP/1.1 protocol for them. + +It's implemented using the ``httpx`` library and needs it to be installed. + +If you want to use this handler you need to replace the default ones for the +``http`` and ``https`` schemes: + +.. code-block:: python + + DOWNLOAD_HANDLERS = { + "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler", + "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler", + } + +.. warning:: + + This handler is experimental, and not yet recommended for production + environments. Future Scrapy versions may introduce related changes without + a deprecation period or warning or even remove it altogether. + +.. note:: + + As this handler is based on a different HTTP client implementation compared + to :class:`~.HTTP11DownloadHandler`, it's expected that its behavior on + some websites may be different. Additionally, these are the Scrapy features + that are explicitly not supported when using it: + + - Proxy support (the :reqmeta:`proxy` meta key). + + - Per-request bind address support (the :reqmeta:`bindaddress` meta key). + The global :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported but the + port number, if specified, will be ignored. + + - The :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and + :setting:`DOWNLOADER_CLIENT_TLS_METHOD` settings. + + - Settings specific to the Twisted networking or HTTP implementation, like + :setting:`DNS_RESOLVER`. + + - Using :ref:`non-asyncio reactors ` (``httpx`` requires + ``asyncio``). + S3DownloadHandler ----------------- diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 1a4f91d93..64722f0e3 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -726,7 +726,7 @@ HttpProxyMiddleware .. class:: HttpProxyMiddleware This middleware sets the HTTP proxy to use for requests, by setting the - ``proxy`` meta value for :class:`~scrapy.Request` objects. + :reqmeta:`proxy` meta value for :class:`~scrapy.Request` objects. Like the Python standard library module :mod:`urllib.request`, it obeys the following environment variables: @@ -735,11 +735,18 @@ HttpProxyMiddleware * ``https_proxy`` * ``no_proxy`` - You can also set the meta key ``proxy`` per-request, to a value like + You can also set the meta key :reqmeta:`proxy` per-request, to a value like ``http://some_proxy_server:port`` or ``http://username:password@some_proxy_server:port``. Keep in mind this value will take precedence over ``http_proxy``/``https_proxy`` environment variables, and it will also ignore ``no_proxy`` environment variable. +.. note:: + + Handling of this meta key needs to be implemented inside the :ref:`download + handler `, so it's not guaranteed to be supported + by all 3rd-party handlers. It's currently unsupported by + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. + HttpProxyMiddleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 5acabb0de..f23c7611c 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -646,7 +646,7 @@ bindaddress The default local outgoing address for download-handler connections. -This setting can be either: +This meta value can be either: - a host address as a string (e.g. ``"127.0.0.2"``), in which case the local port is chosen automatically, or @@ -675,6 +675,10 @@ If not set, built-in HTTP download handlers use the value of Set the :reqmeta:`bindaddress` request meta key to override it for a specific request. +This meta key is not supported by +:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the +:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it. + .. reqmeta:: download_timeout download_timeout diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 6e8d0b54f..c004c253e 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -714,12 +714,6 @@ or even enable client-side authentication (and various other things). ``'scrapy.core.downloader.contextfactory.BrowserLikeContextFactory'``, which uses the platform's certificates to validate remote endpoints. -If you do use a custom ContextFactory, make sure its ``__init__`` method -accepts a ``method`` parameter (this is the ``OpenSSL.SSL`` method mapping -:setting:`DOWNLOADER_CLIENT_TLS_METHOD`), a ``tls_verbose_logging`` -parameter (``bool``) and a ``tls_ciphers`` parameter (see -:setting:`DOWNLOADER_CLIENT_TLS_CIPHERS`). - .. note:: This setting is specific to the built-in Twisted-based download handlers: @@ -753,6 +747,8 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it. (:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and :class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class. + It's currently unsupported by + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. .. setting:: DOWNLOADER_CLIENT_TLS_METHOD @@ -783,6 +779,8 @@ This setting must be one of these string values: (:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and :class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class. + It's currently unsupported by + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. .. setting:: DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING @@ -930,6 +928,13 @@ If set, built-in HTTP download handlers use this value by default. Set the :reqmeta:`bindaddress` request meta key to override it for a specific request. +.. note:: + + Handling of this setting needs to be implemented inside the :ref:`download + handler `, so it's not guaranteed to be supported + by all 3rd-party handlers. Specifying the port is unsupported by + :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. + .. setting:: DOWNLOAD_HANDLERS DOWNLOAD_HANDLERS diff --git a/scrapy/core/downloader/handlers/_httpx.py b/scrapy/core/downloader/handlers/_httpx.py index 916937152..b19a66696 100644 --- a/scrapy/core/downloader/handlers/_httpx.py +++ b/scrapy/core/downloader/handlers/_httpx.py @@ -9,8 +9,6 @@ from http.cookiejar import Cookie, CookieJar from io import BytesIO from typing import TYPE_CHECKING, Any, NoReturn, TypedDict -import httpx - from scrapy import Request, signals from scrapy.exceptions import ( CannotResolveHostError, @@ -46,6 +44,11 @@ if TYPE_CHECKING: from scrapy.crawler import Crawler +try: + import httpx +except ImportError: + httpx = None # type: ignore[assignment] + logger = logging.getLogger(__name__) @@ -81,6 +84,10 @@ class HttpxDownloadHandler(BaseHttpDownloadHandler): f" TWISTED_ENABLED setting. See the asyncio documentation" f" of Scrapy for more information." ) + if httpx is None: # pragma: no cover + raise NotConfigured( + f"{type(self).__name__} requires the httpx library to be installed." + ) super().__init__(crawler) logger.warning( "HttpxDownloadHandler is experimental and is not recommented for production use."