2.6 KiB
Built-in downloader middleware reference
This page describes all downloader middleware components that come with Scrapy. For information on how to use them and how to write your own downloader middleware, see the :ref:`downloader middleware usage guide <topics-downloader-middleware>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".For a list of the components enabled by default (and their orders) see the :setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 12); backlink
Unknown interpreted text role "setting".Available downloader middlewares
DefaultHeadersMiddleware
System Message: ERROR/3 (<stdin>, line 21)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.defaultheaders :synopsis: Default Headers Downloader Middleware
This middleware sets all default requests headers specified in the :setting:`DEFAULT_REQUEST_HEADERS` setting.
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "setting".DebugMiddleware
System Message: ERROR/3 (<stdin>, line 32)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.debug :synopsis: Downloader middlewares for debugging
This is a convenient middleware to inspect what's passing through the downloader middleware. It logs all requests and responses catched by the middleware component methods. This middleware does not use any settings and does not come enabled by default. Instead, it's meant to be inserted at the point of the middleware that you want to inspect.
HttpCacheMiddleware
System Message: ERROR/3 (<stdin>, line 46)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.httpcache :synopsis: HTTP Cache downloader middleware
This middleware provides low-level cache to all HTTP requests and responses. Every request and its corresponding response are cached and then, when that same request is seen again, the response is returned without transferring anything from the Internet.
The HTTP cache is useful for testing spiders faster (without having to wait for downloads every time) and for trying your spider off-line when you don't have an Internet connection.
The :class:`HttpCacheMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "class".
:setting:`HTTPCACHE_DIR` - this one actually enables the cache besides settings the cache dir
System Message: ERROR/3 (<stdin>, line 63); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_IGNORE_MISSING` - ignoring missing requests instead of downloading them
System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_SECTORIZE` - split HTTP cache in several directories (for performance reasons)
System Message: ERROR/3 (<stdin>, line 67); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is considered out of date
System Message: ERROR/3 (<stdin>, line 69); backlink
Unknown interpreted text role "setting".