From 044318920a463d2e04efe2a4c65d8f72d1b5ecb6 Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Sun, 17 Mar 2019 16:54:28 +0530 Subject: [PATCH] doc for creating custom cache storage backend. --- docs/topics/downloader-middleware.rst | 53 +++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0d976077b..f913b059d 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -496,6 +496,59 @@ In order to use this storage backend: .. _LevelDB: https://github.com/google/leveldb .. _leveldb python bindings: https://pypi.python.org/pypi/leveldb +.. _httpcache-storage-custom: + +Implementing custom cache storage backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can implement custom cache storage backend by creating a Python class that +defines the methods described below. + +.. module:: scrapy.extensions.httpcache + +.. class:: CacheStorage + + .. method:: open_spider(spider) + + This method gets called after a spider has been opened for crawling. + + :param spider: the spider which has been opened + :type spider: :class:`~scrapy.spiders.Spider` object + + .. method:: close_spider(spider) + + This method gets called after a spider has been closed. + + :param spider: the spider which has been closed + :type spider: :class:`~scrapy.spiders.Spider` object + + .. method:: retrieve_response(spider, request) + + Returns response if present in cache, or ``None`` otherwise. + + :param spider: the spider which generated the request + :type spider: :class:`~scrapy.spiders.Spider` object + + :param request: the request to find cached reponse for + :type request: :class:`~scrapy.http.Request` object + + .. method:: store_response(spider, request, response) + + Stores the given response in the cache. + + :param spider: the spider for which the response is intended + :type spider: :class:`~scrapy.spiders.Spider` object + + :param request: corresponding request the spider generated + :type request: :class:`~scrapy.http.Request` object + + :param response: the response to store in the cache + :type response: :class:`~scrapy.http.Response` object + +In order to use your storage backend, set: + +* :setting:`HTTPCACHE_STORAGE` to path of your custom storage class. + HTTPCache middleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~