RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update downloader middleware documentation

This commit is contained in:
Pedro Faustino 2013-01-06 18:53:14 +00:00
parent 59dc71f394
commit 5d3a4d755f
1 changed files with 32 additions and 66 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

98
docs/topics/downloader-middleware.rst
View File

@ -293,44 +293,46 @@ HttpCacheMiddleware
.. class:: HttpCacheMiddleware
There are two types of caches:
This middleware provides low-level cache to all HTTP requests and responses.
It has to be combined with a cache storage backend as well as a cache policy.
* Dummy cache
Scrapy ships with two HTTP cache storage backends:
This middleware was designed as a dummy low-level cache to all HTTP
requests and responses, with no awareness of any HTTP Cache-Control
* :ref:`httpcache-dbm-backend`
* :ref:`httpcache-fs-backend`
You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE`
setting. Or you can also implement your own storage backend.
Scrapy ships with two HTTP cache policies:
* Dummy policy
This policy has no awareness of any HTTP Cache-Control
directives. Every request and its corresponding response are cached.
When the same request is seen again, the response is returned
without transferring anything from the Internet.
The HTTP cache is useful for testing spiders faster (without having to
The Dummy policy is useful for testing spiders faster (without having to
wait for downloads every time) and for trying your spider offline, when
an Internet connection is not available. The goal is to be able to
"replay" a spider run *exactly as it run before* and not to use HTTP
caching (to save bandwidth and speed up the crawl).
"replay" a spider run *exactly as it ran before*.
Scrapy ships with two storage backends for the dummy HTTP cache middleware:
* :ref:`httpcache-dbm-backend`
* :ref:`httpcache-fs-backend`
This is the default cache policy.
* Real HTTP cache
* RFC2616 policy
This middleware was designed as a real HTTP cache with HTTP Cache-Control
awareness, aimed at production and used in continuous runs to avoid
downloading unmodified data (to save bandwidth and speed up crawls).
This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
Cache-Control awareness, aimed at production and used in continuous
runs to avoid downloading unmodified data (to save bandwidth and speed up crawls).
In order to use the real HTTP cache, set:
In order to use this policy, set:
* :setting:`HTTPCACHE_USE_DUMMY` to ``False``
* :setting:`HTTPCACHE_STORAGE` to ``'scrapy.contrib.httpcache.DbmRealCacheStorage'``
* :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.downloadermiddleware.httpcache.RFC2616Policy``
Scrapy ships with one storage backend for the real HTTP cache middleware:
* :ref:`httprealcache-dbm-backend`
You can change the storage backend with the :setting:`HTTPCACHE_STORAGE`
setting. Or you can also implement your own backend.
You can change the HTTP cache policy with the :setting:`HTTPCACHE_POLICY`
setting. Or you can also implement your own policy.
.. _httpcache-dbm-backend:
@ -346,20 +348,6 @@ to ``scrapy.contrib.httpcache.DbmCacheStorage``.
By default, it uses the anydbm_ module, but you can change it with the
:setting:`HTTPCACHE_DBM_MODULE` setting.
.. _httprealcache-dbm-backend:
DBM storage backend (real HTTP cache)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It inherits from both ``scrapy.contrib.httpcache.DbmCacheStorage`` and
``scrapy.contrib.httpcache.BaseRealCacheStorage``, the latter providing
HTTP Cache-Control awareness. To use it set :setting:`HTTPCACHE_STORAGE`
to ``scrapy.contrib.httpcache.DbmRealCacheStorage`` and
:setting:`HTTPCACHE_USE_DUMMY` to ``False``.
If you need to create your own storage backend with HTTP Cache-Control awareness,
you can inherit from ``scrapy.contrib.httpcache.BaseRealCacheStorage``.
.. _httpcache-fs-backend:
File system backend
@ -408,15 +396,6 @@ Whether the HTTP cache will be enabled.
.. versionchanged:: 0.11
Before 0.11, :setting:`HTTPCACHE_DIR` was used to enable cache.
.. setting:: HTTPCACHE_USE_DUMMY
HTTPCACHE_USE_DUMMY
^^^^^^^^^^^^^^^^^^^
Default: ``True``
Whether to use the dummy or the real HTTP cache. The default is set to ``True`` for backwards compatibility.
.. setting:: HTTPCACHE_EXPIRATION_SECS
HTTPCACHE_EXPIRATION_SECS
@ -495,29 +474,16 @@ Default: ``'anydbm'``
The database module to use in the :ref:`DBM storage backend
<httpcache-dbm-backend>`. This setting is specific to the DBM backend.
HTTPCACHE_POLICY_REQUEST
^^^^^^^^^^^^^^^^^^^^^^^^
.. setting:: HTTPCACHE_POLICY
HTTPCACHE_POLICY
^^^^^^^^^^^^^^^^
.. versionadded:: 0.18
Default: ```lambda request: True```
Default: ``'scrapy.contrib.downloadermiddleware.httpcache.DummyPolicy'``
A callback function used by the HTTP cache to decide whether a request
is cacheable. The function should take a :class:`~scrapy.http.Request`
object as a parameter and return ``True`` if a cached response can be returned;
or ``False`` if it should be fetched again.
HTTPCACHE_POLICY_RESPONSE
^^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 0.18
Default: ```lambda response: True```
A callback function used by the HTTP cache to decide whether a response
is cacheable. The function should take a :class:`~scrapy.http.Response`
object as a parameter and return ``True`` if a response can be cached;
or ``False`` if it should not be stored in the cache.
The class which implements the cache policy.
HttpCompressionMiddleware
-------------------------