Docs for HttpxDownloadHandler, handle httpx not being installed (#7368)

* Add docs for HttpxDownloadHandler.

* Handle httpx not being installed.

* Revert test_not_configured_without_asyncio().

* Restore the comment.

* Cleanup.

docs/topics/download-handlers.rst

@@ -206,6 +206,57 @@ uses the HTTP/1.1 protocol for them.
 
 It's implemented using :mod:`twisted.web.client`.
 
+HttpxDownloadHandler
+--------------------
+
+.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler
+
+| Supported schemes: ``http``, ``https``.
+| Lazy: no.
+
+This handler supports ``http://host/path`` and ``https://host/path`` URLs and
+uses the HTTP/1.1 protocol for them.
+
+It's implemented using the ``httpx`` library and needs it to be installed.
+
+If you want to use this handler you need to replace the default ones for the
+``http`` and ``https`` schemes:
+
+.. code-block:: python
+
+    DOWNLOAD_HANDLERS = {
+        "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+        "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+    }
+
+.. warning::
+
+    This handler is experimental, and not yet recommended for production
+    environments. Future Scrapy versions may introduce related changes without
+    a deprecation period or warning or even remove it altogether.
+
+.. note::
+
+    As this handler is based on a different HTTP client implementation compared
+    to :class:`~.HTTP11DownloadHandler`, it's expected that its behavior on
+    some websites may be different. Additionally, these are the Scrapy features
+    that are explicitly not supported when using it:
+
+    - Proxy support (the :reqmeta:`proxy` meta key).
+
+    - Per-request bind address support (the :reqmeta:`bindaddress` meta key).
+      The global :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported but the
+      port number, if specified, will be ignored.
+
+    - The :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
+      :setting:`DOWNLOADER_CLIENT_TLS_METHOD` settings.
+
+    - Settings specific to the Twisted networking or HTTP implementation, like
+      :setting:`DNS_RESOLVER`.
+
+    - Using :ref:`non-asyncio reactors <disable-asyncio>` (``httpx`` requires
+      ``asyncio``).
+
 S3DownloadHandler
 -----------------
 

docs/topics/downloader-middleware.rst

@@ -726,7 +726,7 @@ HttpProxyMiddleware
 .. class:: HttpProxyMiddleware
 
    This middleware sets the HTTP proxy to use for requests, by setting the
-   ``proxy`` meta value for :class:`~scrapy.Request` objects.
+   :reqmeta:`proxy` meta value for :class:`~scrapy.Request` objects.
 
    Like the Python standard library module :mod:`urllib.request`, it obeys
    the following environment variables:
@@ -735,11 +735,18 @@ HttpProxyMiddleware
    * ``https_proxy``
    * ``no_proxy``
 
-   You can also set the meta key ``proxy`` per-request, to a value like
+   You can also set the meta key :reqmeta:`proxy` per-request, to a value like
    ``http://some_proxy_server:port`` or ``http://username:password@some_proxy_server:port``.
    Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
    environment variables, and it will also ignore ``no_proxy`` environment variable.
 
+.. note::
+
+    Handling of this meta key needs to be implemented inside the :ref:`download
+    handler <topics-download-handlers>`, so it's not guaranteed to be supported
+    by all 3rd-party handlers. It's currently unsupported by
+    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
+
 HttpProxyMiddleware settings
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/topics/request-response.rst

@@ -646,7 +646,7 @@ bindaddress
 
 The default local outgoing address for download-handler connections.
 
-This setting can be either:
+This meta value can be either:
 
 - a host address as a string (e.g. ``"127.0.0.2"``), in which case the local
   port is chosen automatically, or
@@ -675,6 +675,10 @@ If not set, built-in HTTP download handlers use the value of
 Set the :reqmeta:`bindaddress` request meta key to override it for a
 specific request.
 
+This meta key is not supported by
+:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the
+:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it.
+
 .. reqmeta:: download_timeout
 
 download_timeout

docs/topics/settings.rst

@@ -714,12 +714,6 @@ or even enable client-side authentication (and various other things).
     ``'scrapy.core.downloader.contextfactory.BrowserLikeContextFactory'``,
     which uses the platform's certificates to validate remote endpoints.
 
-If you do use a custom ContextFactory, make sure its ``__init__`` method
-accepts a ``method`` parameter (this is the ``OpenSSL.SSL`` method mapping
-:setting:`DOWNLOADER_CLIENT_TLS_METHOD`), a ``tls_verbose_logging``
-parameter (``bool``) and a ``tls_ciphers`` parameter (see
-:setting:`DOWNLOADER_CLIENT_TLS_CIPHERS`).
-
 .. note::
 
     This setting is specific to the built-in Twisted-based download handlers:
@@ -753,6 +747,8 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it.
     (:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
     :class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs
     to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.
+    It's currently unsupported by
+    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
 
 .. setting:: DOWNLOADER_CLIENT_TLS_METHOD
 
@@ -783,6 +779,8 @@ This setting must be one of these string values:
     (:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
     :class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs
     to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.
+    It's currently unsupported by
+    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
 
 .. setting:: DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING
 
@@ -930,6 +928,13 @@ If set, built-in HTTP download handlers use this value by default.
 Set the :reqmeta:`bindaddress` request meta key to override it for a specific
 request.
 
+.. note::
+
+    Handling of this setting needs to be implemented inside the :ref:`download
+    handler <topics-download-handlers>`, so it's not guaranteed to be supported
+    by all 3rd-party handlers. Specifying the port is unsupported by
+    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
+
 .. setting:: DOWNLOAD_HANDLERS
 
 DOWNLOAD_HANDLERS

scrapy/core/downloader/handlers/_httpx.py

@@ -9,8 +9,6 @@ from http.cookiejar import Cookie, CookieJar
 from io import BytesIO
 from typing import TYPE_CHECKING, Any, NoReturn, TypedDict
 
-import httpx
-
 from scrapy import Request, signals
 from scrapy.exceptions import (
     CannotResolveHostError,
@@ -46,6 +44,11 @@ if TYPE_CHECKING:
     from scrapy.crawler import Crawler
 
 
+try:
+    import httpx
+except ImportError:
+    httpx = None  # type: ignore[assignment]
+
 logger = logging.getLogger(__name__)
 
 
@@ -81,6 +84,10 @@ class HttpxDownloadHandler(BaseHttpDownloadHandler):
                 f" TWISTED_ENABLED setting. See the asyncio documentation"
                 f" of Scrapy for more information."
             )
+        if httpx is None:  # pragma: no cover
+            raise NotConfigured(
+                f"{type(self).__name__} requires the httpx library to be installed."
+            )
         super().__init__(crawler)
         logger.warning(
             "HttpxDownloadHandler is experimental and is not recommented for production use."