diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 48292e10c..a47ffa83e 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -293,44 +293,46 @@ HttpCacheMiddleware .. class:: HttpCacheMiddleware - There are two types of caches: + This middleware provides low-level cache to all HTTP requests and responses. + It has to be combined with a cache storage backend as well as a cache policy. - * Dummy cache + Scrapy ships with two HTTP cache storage backends: - This middleware was designed as a dummy low-level cache to all HTTP - requests and responses, with no awareness of any HTTP Cache-Control + * :ref:`httpcache-dbm-backend` + * :ref:`httpcache-fs-backend` + + You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE` + setting. Or you can also implement your own storage backend. + + + Scrapy ships with two HTTP cache policies: + + * Dummy policy + + This policy has no awareness of any HTTP Cache-Control directives. Every request and its corresponding response are cached. When the same request is seen again, the response is returned without transferring anything from the Internet. - The HTTP cache is useful for testing spiders faster (without having to + The Dummy policy is useful for testing spiders faster (without having to wait for downloads every time) and for trying your spider offline, when an Internet connection is not available. The goal is to be able to - "replay" a spider run *exactly as it run before* and not to use HTTP - caching (to save bandwidth and speed up the crawl). + "replay" a spider run *exactly as it ran before*. - Scrapy ships with two storage backends for the dummy HTTP cache middleware: - - * :ref:`httpcache-dbm-backend` - * :ref:`httpcache-fs-backend` + This is the default cache policy. - * Real HTTP cache + * RFC2616 policy - This middleware was designed as a real HTTP cache with HTTP Cache-Control - awareness, aimed at production and used in continuous runs to avoid - downloading unmodified data (to save bandwidth and speed up crawls). + This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP + Cache-Control awareness, aimed at production and used in continuous + runs to avoid downloading unmodified data (to save bandwidth and speed up crawls). - In order to use the real HTTP cache, set: + In order to use this policy, set: - * :setting:`HTTPCACHE_USE_DUMMY` to ``False`` - * :setting:`HTTPCACHE_STORAGE` to ``'scrapy.contrib.httpcache.DbmRealCacheStorage'`` + * :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.downloadermiddleware.httpcache.RFC2616Policy`` - Scrapy ships with one storage backend for the real HTTP cache middleware: - - * :ref:`httprealcache-dbm-backend` - - You can change the storage backend with the :setting:`HTTPCACHE_STORAGE` - setting. Or you can also implement your own backend. + You can change the HTTP cache policy with the :setting:`HTTPCACHE_POLICY` + setting. Or you can also implement your own policy. .. _httpcache-dbm-backend: @@ -346,20 +348,6 @@ to ``scrapy.contrib.httpcache.DbmCacheStorage``. By default, it uses the anydbm_ module, but you can change it with the :setting:`HTTPCACHE_DBM_MODULE` setting. -.. _httprealcache-dbm-backend: - -DBM storage backend (real HTTP cache) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It inherits from both ``scrapy.contrib.httpcache.DbmCacheStorage`` and -``scrapy.contrib.httpcache.BaseRealCacheStorage``, the latter providing -HTTP Cache-Control awareness. To use it set :setting:`HTTPCACHE_STORAGE` -to ``scrapy.contrib.httpcache.DbmRealCacheStorage`` and -:setting:`HTTPCACHE_USE_DUMMY` to ``False``. - -If you need to create your own storage backend with HTTP Cache-Control awareness, -you can inherit from ``scrapy.contrib.httpcache.BaseRealCacheStorage``. - .. _httpcache-fs-backend: File system backend @@ -408,15 +396,6 @@ Whether the HTTP cache will be enabled. .. versionchanged:: 0.11 Before 0.11, :setting:`HTTPCACHE_DIR` was used to enable cache. -.. setting:: HTTPCACHE_USE_DUMMY - -HTTPCACHE_USE_DUMMY -^^^^^^^^^^^^^^^^^^^ - -Default: ``True`` - -Whether to use the dummy or the real HTTP cache. The default is set to ``True`` for backwards compatibility. - .. setting:: HTTPCACHE_EXPIRATION_SECS HTTPCACHE_EXPIRATION_SECS @@ -495,29 +474,16 @@ Default: ``'anydbm'`` The database module to use in the :ref:`DBM storage backend `. This setting is specific to the DBM backend. -HTTPCACHE_POLICY_REQUEST -^^^^^^^^^^^^^^^^^^^^^^^^ +.. setting:: HTTPCACHE_POLICY + +HTTPCACHE_POLICY +^^^^^^^^^^^^^^^^ .. versionadded:: 0.18 -Default: ```lambda request: True``` +Default: ``'scrapy.contrib.downloadermiddleware.httpcache.DummyPolicy'`` -A callback function used by the HTTP cache to decide whether a request -is cacheable. The function should take a :class:`~scrapy.http.Request` -object as a parameter and return ``True`` if a cached response can be returned; -or ``False`` if it should be fetched again. - -HTTPCACHE_POLICY_RESPONSE -^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. versionadded:: 0.18 - -Default: ```lambda response: True``` - -A callback function used by the HTTP cache to decide whether a response -is cacheable. The function should take a :class:`~scrapy.http.Response` -object as a parameter and return ``True`` if a response can be cached; -or ``False`` if it should not be stored in the cache. +The class which implements the cache policy. HttpCompressionMiddleware -------------------------