9.9 KiB

Raw Blame History

Download handlers

Download handlers are Scrapy :ref:`components <topics-components>` used to download :ref:`requests <topics-request-response>` and produce responses from them.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Using download handlers

The :setting:`DOWNLOAD_HANDLERS_BASE` and :setting:`DOWNLOAD_HANDLERS` settings tell Scrapy which handler is responsible for a given URL scheme. Their values are merged into a mapping from scheme names to handler classes. When Scrapy initializes it creates instances of all configured download handlers (except for :ref:`lazy ones <lazy-download-handlers>`) and stores them in a similar mapping. When Scrapy needs to download a request it extracts the scheme from its URL, finds the handler for this scheme, passes the request to it and gets a response from it. If there is no handler for the scheme, the request is not downloaded and a :exc:`~scrapy.exceptions.NotSupported` exception is raised.

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 14); backlink

Unknown interpreted text role "exc".

The :setting:`DOWNLOAD_HANDLERS_BASE` setting contains the default mapping of handlers. You can use the :setting:`DOWNLOAD_HANDLERS` setting to add handlers for additional schemes and to replace or disable default ones:

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "setting".

System Message: WARNING/2 (<stdin>, line 28)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    DOWNLOAD_HANDLERS = {
        # disable support for ftp:// requests
        "ftp": None,
        # replace the default one for http://
        "http": "my.download_handlers.HttpHandler",
        # http:// and https:// are different schemes,
        # even though they may use the same handler
        "https": "my.download_handlers.HttpHandler",
        # support for any custom scheme can be added
        "sftp": "my.download_handlers.SftpHandler",
    }

Replacing HTTP(S) download handlers

While Scrapy provides a default handler for http and https schemes, users may want to use a different handler, provided by Scrapy or by some 3rd-party package. There are several considerations to keep in mind related to this.

First of all, as http and https are separate schemes, they need separate entries in the :setting:`DOWNLOAD_HANDLERS` setting, even though it's likely that the same handler class will be used for both schemes.

System Message: ERROR/3 (<stdin>, line 50); backlink

Unknown interpreted text role "setting".

Additionally, some of the Scrapy settings, like :setting:`DOWNLOAD_MAXSIZE`, are honored by the default HTTP(S) handler but not necessarily by alternative ones. The same may apply to other Scrapy features, e.g. the :signal:`bytes_received` and :signal:`headers_received` signals.

System Message: ERROR/3 (<stdin>, line 54); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 54); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 54); backlink

Unknown interpreted text role "signal".

Lazy instantiation of download handlers

A download handler can be marked as "lazy" by setting its lazy class attribute to True. Such handlers are only instantiated when they need to download their first request. This may be useful when the instantiation is slow or requires dependencies that are not always available, and the handler is not needed on every spider run. For example, :class:`the built-in S3 handler <.S3DownloadHandler>` is lazy.

System Message: ERROR/3 (<stdin>, line 64); backlink

Unknown interpreted text role "class".

Writing your own download handler

A download handler is a :ref:`component <topics-components>` that defines the following API:

System Message: ERROR/3 (<stdin>, line 74); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 79)

Unknown directive type "attribute".

.. attribute:: lazy
    :type: bool

    If ``False``, the handler will be instantiated when Scrapy is
    initialized.

    If ``True``, the handler will only be instantiated when the first
    request handled by it needs to be downloaded.

System Message: ERROR/3 (<stdin>, line 88)

Unknown directive type "method".

.. method:: download_request(request: Request) -> Response:
    :async:

    Download the given request and return a response.

System Message: ERROR/3 (<stdin>, line 93)

Unknown directive type "method".

.. method:: close() -> None
    :async:

    Clean up any resources used by the handler.

An optional base class for custom handlers is provided:

System Message: ERROR/3 (<stdin>, line 100)

Unknown directive type "autoclass".

.. autoclass:: scrapy.core.downloader.handlers.base.BaseDownloadHandler
    :members:
    :undoc-members:
    :member-order: bysource

Exceptions raised by download handlers

System Message: ERROR/3 (<stdin>, line 110)

Unknown directive type "versionadded".

.. versionadded:: 2.15.0

The built-in download handlers raise Scrapy-specific exceptions instead of implementation-specific ones, so that code that handles these exceptions can be written in a generic way. We recommend custom download handlers to also use these exceptions.

System Message: ERROR/3 (<stdin>, line 117)