2.8 KiB
Built-in downloader middleware reference
This document explains all downloader middleware components that come with Scrapy. For information on how to use them and how to write your own downloader middleware, see the :ref:`downloader middleware usage guide <topics-downloader-middleware>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Available downloader middlewares
"Common" downloader middleware
System Message: ERROR/3 (<stdin>, line 20)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.common :synopsis: Downloader middleware for performing basic required tasks
This middleware performs some commonly required tasks over all requests, and thus it's recommended to leave it always enabled. Those tasks are:
If the Accept request header is not already set, then set it to :setting:`REQUEST_HEADER_ACCEPT`
System Message: ERROR/3 (<stdin>, line 28); backlink
Unknown interpreted text role "setting".
If the Accept-Language request header is not already set, then set it to :setting:`REQUEST_HEADER_ACCEPT_LANGUAGE`
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "setting".
If the request method is POST and the Content-Type header is not set, then set it to 'application/x-www-form-urlencoded', the default Form content type.
If the request contains a body and the Content-Length headers it not set, then set it to the len(body).
HttpCacheMiddleware
System Message: ERROR/3 (<stdin>, line 46)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.httpcache :synopsis: HTTP Cache downloader middleware
This middleware provides low-level cache to all HTTP requests and responses. Every request and its corresponding response are cached and then, when that same request is seen again, the response is returned without transferring anything from the Internet.
The HTTP cache is useful for testing spiders faster (without having to wait for downloads every time) and for trying your spider off-line when you don't have an Internet connection.
The :class:`HttpCacheMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "class".
:setting:`HTTPCACHE_DIR` - this one actually enables the cache besides settings the cache dir
System Message: ERROR/3 (<stdin>, line 63); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_IGNORE_MISSING` - ignoring missing requests instead of downloading them
System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_SECTORIZE` - split HTTP cache in several directories (for performance reasons)
System Message: ERROR/3 (<stdin>, line 67); backlink
Unknown interpreted text role "setting".
:setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is considered out of date
System Message: ERROR/3 (<stdin>, line 69); backlink
Unknown interpreted text role "setting".