RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

revert default HTTP cache policy to dummy (instead of RFC2616)

This commit is contained in:
Pablo Hoffman 2013-01-17 13:08:29 -02:00
parent 897195186a
commit c31441a273
2 changed files with 24 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
docs/topics/downloader-middleware.rst
View File

@ -314,10 +314,30 @@ HttpCacheMiddleware
setting. Or you can also implement your own policy.
.. _httpcache-policy-dummy:
Dummy policy (default)
~~~~~~~~~~~~~~~~~~~~~~
This policy has no awareness of any HTTP Cache-Control directives.
Every request and its corresponding response are cached. When the same
request is seen again, the response is returned without transferring
anything from the Internet.
The Dummy policy is useful for testing spiders faster (without having
to wait for downloads every time) and for trying your spider offline,
when an Internet connection is not available. The goal is to be able to
"replay" a spider run *exactly as it ran before*.
In order to use this policy, set:
* :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.httpcache.DummyPolicy``
.. _httpcache-policy-rfc2616:
RFC2616 policy (default)
~~~~~~~~~~~~~~~~~~~~~~~~
RFC2616 policy
~~~~~~~~~~~~~~
This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP
Cache-Control awareness, aimed at production and used in continuous
@ -350,26 +370,6 @@ In order to use this policy, set:
This is the default cache policy.
.. _httpcache-policy-dummy:
Dummy policy
~~~~~~~~~~~~
This policy has no awareness of any HTTP Cache-Control directives.
Every request and its corresponding response are cached. When the same
request is seen again, the response is returned without transferring
anything from the Internet.
The Dummy policy is useful for testing spiders faster (without having
to wait for downloads every time) and for trying your spider offline,
when an Internet connection is not available. The goal is to be able to
"replay" a spider run *exactly as it ran before*.
In order to use this policy, set:
* :setting:`HTTPCACHE_POLICY` to ``scrapy.contrib.httpcache.DummyPolicy``
.. _httpcache-storage-dbm:
DBM storage backend (default)
@ -523,7 +523,7 @@ HTTPCACHE_POLICY
.. versionadded:: 0.18
Default: ``'scrapy.contrib.httpcache.RFC2616Policy'``
Default: ``'scrapy.contrib.httpcache.DummyPolicy'``
The class which implements the cache policy.

2
scrapy/settings/default_settings.py
View File

@ -143,7 +143,7 @@ HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_IGNORE_SCHEMES = ['file']
HTTPCACHE_DBM_MODULE = 'anydbm'
HTTPCACHE_POLICY = 'scrapy.contrib.httpcache.RFC2616Policy'
HTTPCACHE_POLICY = 'scrapy.contrib.httpcache.DummyPolicy'
ITEM_PROCESSOR = 'scrapy.contrib.pipeline.ItemPipelineManager'