1.5 KiB
Built-in downloader middleware reference
This document explains all downloader middleware components that come with Scrapy. For information on how to use them and how to write your own downloader middleware, see the :ref:`downloader middleware usage guide <topics-downloader-middleware>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Available downloader middlewares
"Common" downloader middleware
System Message: ERROR/3 (<stdin>, line 20)
Unknown directive type "module".
.. module:: scrapy.contrib.downloadermiddleware.common :synopsis: Downloader middleware for performing basic required tasks
This middleware performs some commonly required tasks over all requests, and thus it's recommended to leave it always enabled. Those tasks are:
If the Accept request header is not already set, then set it to :setting:`REQUEST_HEADER_ACCEPT`
System Message: ERROR/3 (<stdin>, line 28); backlink
Unknown interpreted text role "setting".
If the Accept-Language request header is not already set, then set it to :setting:`REQUEST_HEADER_ACCEPT_LANGUAGE`
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "setting".
If the request method is POST and the Content-Type header is not set, then set it to 'application/x-www-form-urlencoded', the default Form content type.
If the request contains a body and the Content-Length headers it not set, then set it to the len(body).