RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/download-handlers.rst

14 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Download handlers

Download handlers are Scrapy :ref:`components <topics-components>` used to download :ref:`requests <topics-request-response>` and produce responses from them.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Using download handlers

The :setting:`DOWNLOAD_HANDLERS_BASE` and :setting:`DOWNLOAD_HANDLERS` settings tell Scrapy which handler is responsible for a given URL scheme. Their values are merged into a mapping from scheme names to handler classes. When Scrapy initializes it creates instances of all configured download handlers (except for :ref:`lazy ones <lazy-download-handlers>`) and stores them in a similar mapping. When Scrapy needs to download a request it extracts the scheme from its URL, finds the handler for this scheme, passes the request to it and gets a response from it. If there is no handler for the scheme, the request is not downloaded and a :exc:`~scrapy.exceptions.NotSupported` exception is raised.

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "exc".

The :setting:`DOWNLOAD_HANDLERS_BASE` setting contains the default mapping of handlers. You can use the :setting:`DOWNLOAD_HANDLERS` setting to add handlers for additional schemes and to replace or disable default ones:

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "setting".

System Message: WARNING/2 (<stdin>, line 28)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        # disable support for ftp:// requests
        "ftp": None,
        # replace the default one for http://
        "http": "my.download_handlers.HttpHandler",
        # http:// and https:// are different schemes,
        # even though they may use the same handler
        "https": "my.download_handlers.HttpHandler",
        # support for any custom scheme can be added
        "sftp": "my.download_handlers.SftpHandler",
    }

System Message: ERROR/3 (<stdin>, line 42)

Unknown directive type "seealso".

.. seealso:: :ref:`security-unencrypted-protocols` and
    :ref:`security-local-resources`, for the security implications of the
    default ``http``, ``ftp``, ``file`` and ``data`` handlers.

Replacing HTTP(S) download handlers

While Scrapy provides a default handler for http and https schemes, users may want to use a different handler, provided by Scrapy or by some 3rd-party package. There are several considerations to keep in mind related to this.

First of all, as http and https are separate schemes, they need separate entries in the :setting:`DOWNLOAD_HANDLERS` setting, even though it's likely that the same handler class will be used for both schemes.

System Message: ERROR/3 (<stdin>, line 54); backlink

Unknown interpreted text role "setting".

Additionally, some of the Scrapy settings, like :setting:`DOWNLOAD_MAXSIZE`, are honored by the default HTTP(S) handler but not necessarily by alternative ones. The same may apply to other Scrapy features, e.g. the :signal:`bytes_received` and :signal:`headers_received` signals.

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "signal".

Lazy instantiation of download handlers

A download handler can be marked as "lazy" by setting its lazy class attribute to True. Such handlers are only instantiated when they need to download their first request. This may be useful when the instantiation is slow or requires dependencies that are not always available, and the handler is not needed on every spider run. For example, :class:`the built-in S3 handler <.S3DownloadHandler>` is lazy.

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "class".

Writing your own download handler

A download handler is a :ref:`component <topics-components>` that defines the following API:

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 81)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.DownloadHandlerProtocol
    :members:

An optional base class for custom handlers is provided:

System Message: ERROR/3 (<stdin>, line 86)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.base.BaseDownloadHandler
    :members:
    :undoc-members:
    :exclude-members: close, download_request, lazy

Exceptions raised by download handlers

System Message: ERROR/3 (<stdin>, line 96)

Unknown directive type "versionadded".

.. versionadded:: 2.15.0

The built-in download handlers raise Scrapy-specific exceptions instead of implementation-specific ones, so that code that handles these exceptions can be written in a generic way. We recommend custom download handlers to also use these exceptions.

System Message: ERROR/3 (<stdin>, line 103)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.CannotResolveHostError

System Message: ERROR/3 (<stdin>, line 105)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.DownloadCancelledError

System Message: ERROR/3 (<stdin>, line 107)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.DownloadConnectionRefusedError

System Message: ERROR/3 (<stdin>, line 109)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.DownloadFailedError

System Message: ERROR/3 (<stdin>, line 111)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.DownloadTimeoutError

System Message: ERROR/3 (<stdin>, line 113)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.ResponseDataLossError

System Message: ERROR/3 (<stdin>, line 115)

Unknown directive type "autoexception".

.. autoexception:: scrapy.exceptions.UnsupportedURLSchemeError

Built-in HTTP download handlers reference

Scrapy ships several handlers for HTTP and HTTPS requests. While all of them support basic features, they may differ in support of specific Scrapy features and settings and HTTP protocol features. See the documentation of specific handlers and specific settings for more information. Additionally, as the underlying HTTP client implementations differ between handlers, the behavior of specific websites may be different when doing the same Scrapy requests but using different handlers.

Here is a comparison of some features of the built-in HTTP handlers, see the individual handler docs for more differences:

Feature H2DownloadHandler HTTP11DownloadHandler HttpxDownloadHandler
Requires asyncio No No Yes
Requires a reactor Yes Yes No
HTTP/1.1 No Yes Yes
HTTP/2 Yes No Yes
TLS implementation cryptography cryptography Stdlib ssl
HTTP proxies No Yes Yes
SOCKS proxies No No Yes
Bad header handling Not applicable Skip bad Fail

Bad header handling is what a handler does when a response has a bad header line, e.g. one with no colon in it, which some servers send. Handlers that skip bad header lines, like web browsers do, still parse the header lines that follow them; other handlers also lose those, or cannot download such responses at all.

You can find additional HTTP download handlers in the scrapy-download-handlers-incubator package. This package is made by the Scrapy developers and contains experimental handlers that may be included in some later Scrapy version but can already be used. Please refer to the documentation of this package for more information.

H2DownloadHandler

Note

Requires the :ref:`twisted-http2 <extras>` extra.

System Message: ERROR/3 (<stdin>, line 164); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 166)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.http2.H2DownloadHandler

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 170); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 171); backlink

Unknown interpreted text role "ref".

This handler supports https://host/path URLs and uses the HTTP/2 protocol for them.

It's implemented using :mod:`twisted.web.client` and the h2 library.

System Message: ERROR/3 (<stdin>, line 176); backlink

Unknown interpreted text role "mod".

If you want to use this handler you need to replace the default one for the https scheme:

System Message: WARNING/2 (<stdin>, line 181)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        "https": "scrapy.core.downloader.handlers.http2.H2DownloadHandler",
    }

Features and limitations

Warning

This handler is experimental, and not yet recommended for production environments. Future Scrapy versions may introduce related changes without a deprecation period or warning.

HTTP proxies No (not implemented)
SOCKS proxies No (not supported by the library)
HTTP/2 Yes
Bad header handling Not applicable (HTTP/2 only)
response.certificate

:class:`twisted.internet.ssl.Certificate` object

System Message: ERROR/3 (<stdin>, line 202); backlink

Unknown interpreted text role "class".
Per-request bindaddress Yes
TLS implementation pyOpenSSL/cryptography

Other limitations:

  • No support for HTTP/1.1.

  • IPv6 support requires setting :setting:`TWISTED_DNS_RESOLVER` to scrapy.resolver.CachingHostnameResolver.

    System Message: ERROR/3 (<stdin>, line 210); backlink

    Unknown interpreted text role "setting".

Known limitations of the HTTP/2 support:

  • No support for HTTP/2 Cleartext (h2c), since no major browser supports HTTP/2 unencrypted (refer http2 faq).
  • No setting to specify a maximum frame size larger than the default value, 16384. Connections to servers that send a larger frame will fail.
  • No support for server pushes, which are ignored.

HTTP11DownloadHandler

System Message: ERROR/3 (<stdin>, line 230)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler

System Message: ERROR/3 (<stdin>, line 233); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 234); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 235); backlink

Unknown interpreted text role "ref".

This handler supports http://host/path and https://host/path URLs and uses the HTTP/1.1 protocol for them.

It's implemented using :mod:`twisted.web.client`.

System Message: ERROR/3 (<stdin>, line 240); backlink

Unknown interpreted text role "mod".

Features and limitations

HTTP proxies Yes
SOCKS proxies No (not supported by the library)
HTTP/2 No (implemented as a separate handler)
Bad header handling Skip bad, like web browsers do
response.certificate

:class:`twisted.internet.ssl.Certificate` object

System Message: ERROR/3 (<stdin>, line 251); backlink

Unknown interpreted text role "class".
Per-request bindaddress Yes
TLS implementation pyOpenSSL/cryptography

System Message: ERROR/3 (<stdin>, line 255)

Unknown directive type "versionchanged".

.. versionchanged:: VERSION
   Bad header lines with no colon in them are now skipped, instead of making
   the whole response impossible to download.

Other limitations:

  • IPv6 support requires setting :setting:`TWISTED_DNS_RESOLVER` to scrapy.resolver.CachingHostnameResolver.

    System Message: ERROR/3 (<stdin>, line 261); backlink

    Unknown interpreted text role "setting".

  • HTTPS proxies to HTTPS destinations are not supported.

HttpxDownloadHandler

Note

Requires the :ref:`httpx <extras>` extra.

System Message: ERROR/3 (<stdin>, line 271); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 273)

Unknown directive type "versionadded".

.. versionadded:: 2.15.0

System Message: ERROR/3 (<stdin>, line 275)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler

System Message: ERROR/3 (<stdin>, line 278); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 279); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 280); backlink

Unknown interpreted text role "ref".

This handler supports http://host/path and https://host/path URLs and uses the HTTP/1.1 or HTTP/2 protocol for them.

It's implemented using the httpx2 library.

If you want to use this handler you need to replace the default ones for the http and https schemes:

System Message: WARNING/2 (<stdin>, line 292)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
        "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    }

Features and limitations

Warning

This handler is experimental, and not yet recommended for production environments. Future Scrapy versions may introduce related changes without a deprecation period or warning or even remove it altogether.

HTTP proxies Yes
SOCKS proxies Yes (SOCKS5)
HTTP/2 Yes
Bad header handling Fail (not supported by the library)
response.certificate DER bytes
Per-request bindaddress No (not supported by the library)
TLS implementation Standard library ssl

Other limitations:

  • The handler creates a separate connection pool for each proxy URL (due to limitations of httpx) which may lead to higher resource usage when using proxy rotation.

System Message: ERROR/3 (<stdin>, line 324)

Unknown directive type "setting".

.. setting:: HTTPX_HTTP2_ENABLED

HTTPX_HTTP2_ENABLED

System Message: ERROR/3 (<stdin>, line 329)

Unknown directive type "versionadded".

.. versionadded:: 2.17.0

Default: False

Whether to enable HTTP/2 support in this handler.

Built-in non-HTTP download handlers reference

DataURIDownloadHandler

System Message: ERROR/3 (<stdin>, line 341)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler

System Message: ERROR/3 (<stdin>, line 344); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 345); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 346); backlink

Unknown interpreted text role "ref".

This handler supports RFC 2397 data:content/type;base64, data URIs.

FileDownloadHandler

System Message: ERROR/3 (<stdin>, line 353)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.file.FileDownloadHandler

System Message: ERROR/3 (<stdin>, line 356); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 357); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 358); backlink

Unknown interpreted text role "ref".

This handler supports file:///path local file URIs. It doesn't support remote files.

FTPDownloadHandler

System Message: ERROR/3 (<stdin>, line 366)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.ftp.FTPDownloadHandler

System Message: ERROR/3 (<stdin>, line 369); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 370); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 371); backlink

Unknown interpreted text role "ref".

This handler supports ftp://host/path FTP URIs.

It's implemented using :mod:`twisted.protocols.ftp`.

System Message: ERROR/3 (<stdin>, line 375); backlink

Unknown interpreted text role "mod".

S3DownloadHandler

Note

Requires the :ref:`s3 <extras>` extra.

System Message: ERROR/3 (<stdin>, line 382); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 384)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.s3.S3DownloadHandler

System Message: ERROR/3 (<stdin>, line 387); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 388); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 389); backlink

Unknown interpreted text role "ref".

This handler supports s3://bucket/path S3 URIs.

It's implemented using the botocore library.

</html>