mirror of https://github.com/scrapy/scrapy.git
43 lines
1.5 KiB
ReStructuredText
43 lines
1.5 KiB
ReStructuredText
.. _ref-downloader-middleware:
|
|
|
|
========================================
|
|
Built-in downloader middleware reference
|
|
========================================
|
|
|
|
This document explains all downloader middleware components that come with
|
|
Scrapy. For information on how to use them and how to write your own downloader
|
|
middleware, see the :ref:`downloader middleware usage guide
|
|
<topics-downloader-middleware>`.
|
|
|
|
Available downloader middlewares
|
|
================================
|
|
|
|
.. _ref-downloader-middleware-common:
|
|
|
|
"Common" downloader middleware
|
|
------------------------------
|
|
|
|
.. module:: scrapy.contrib.downloadermiddleware.common
|
|
:synopsis: Downloader middleware for performing basic required tasks
|
|
|
|
.. class:: scrapy.contrib.downloadermiddleware.common.CommonMiddleware
|
|
|
|
This middleware performs some commonly required tasks over all requests, and
|
|
thus it's recommended to leave it always enabled. Those tasks are:
|
|
|
|
* If the ``Accept`` request header is not already set, then set it to
|
|
:setting:`REQUEST_HEADER_ACCEPT`
|
|
|
|
* If the ``Accept-Language`` request header is not already set, then set it
|
|
to :setting:`REQUEST_HEADER_ACCEPT_LANGUAGE`
|
|
|
|
* If the request method is ``POST`` and the ``Content-Type`` header is not
|
|
set, then set it to ``'application/x-www-form-urlencoded'``, the `default
|
|
Form content type`_.
|
|
|
|
* If the request contains a body and the ``Content-Length`` headers it not
|
|
set, then set it to the ``len(body)``.
|
|
|
|
.. _default Form content type: http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1
|
|
|