mirror of https://github.com/scrapy/scrapy.git
72 lines
2.8 KiB
ReStructuredText
72 lines
2.8 KiB
ReStructuredText
.. _ref-downloader-middleware:
|
|
|
|
========================================
|
|
Built-in downloader middleware reference
|
|
========================================
|
|
|
|
This document explains all downloader middleware components that come with
|
|
Scrapy. For information on how to use them and how to write your own downloader
|
|
middleware, see the :ref:`downloader middleware usage guide
|
|
<topics-downloader-middleware>`.
|
|
|
|
Available downloader middlewares
|
|
================================
|
|
|
|
.. _ref-downloader-middleware-common:
|
|
|
|
"Common" downloader middleware
|
|
------------------------------
|
|
|
|
.. module:: scrapy.contrib.downloadermiddleware.common
|
|
:synopsis: Downloader middleware for performing basic required tasks
|
|
|
|
.. class:: scrapy.contrib.downloadermiddleware.common.CommonMiddleware
|
|
|
|
This middleware performs some commonly required tasks over all requests, and
|
|
thus it's recommended to leave it always enabled. Those tasks are:
|
|
|
|
* If the ``Accept`` request header is not already set, then set it to
|
|
:setting:`REQUEST_HEADER_ACCEPT`
|
|
|
|
* If the ``Accept-Language`` request header is not already set, then set it
|
|
to :setting:`REQUEST_HEADER_ACCEPT_LANGUAGE`
|
|
|
|
* If the request method is ``POST`` and the ``Content-Type`` header is not
|
|
set, then set it to ``'application/x-www-form-urlencoded'``, the `default
|
|
Form content type`_.
|
|
|
|
* If the request contains a body and the ``Content-Length`` headers it not
|
|
set, then set it to the ``len(body)``.
|
|
|
|
.. _default Form content type: http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1
|
|
|
|
HttpCacheMiddleware
|
|
-------------------
|
|
|
|
.. module:: scrapy.contrib.downloadermiddleware.httpcache
|
|
:synopsis: HTTP Cache downloader middleware
|
|
|
|
.. class:: HttpCacheMiddleware
|
|
|
|
This middleware provides low-level cache to all HTTP requests and responses.
|
|
Every request and its corresponding response are cached and then, when that
|
|
same request is seen again, the response is returned without transferring
|
|
anything from the Internet.
|
|
|
|
The HTTP cache is useful for testing spiders faster (without having to wait for
|
|
downloads every time) and for trying your spider off-line when you don't have
|
|
an Internet connection.
|
|
|
|
The :class:`HttpCacheMiddleware` can be configured through the following
|
|
settings (see the settings documentation for more info):
|
|
|
|
* :setting:`HTTPCACHE_DIR` - this one actually enables the cache besides
|
|
settings the cache dir
|
|
* :setting:`HTTPCACHE_IGNORE_MISSING` - ignoring missing requests instead
|
|
of downloading them
|
|
* :setting:`HTTPCACHE_SECTORIZE` - split HTTP cache in several directories
|
|
(for performance reasons)
|
|
* :setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is
|
|
considered out of date
|
|
|