From 044318920a463d2e04efe2a4c65d8f72d1b5ecb6 Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Sun, 17 Mar 2019 16:54:28 +0530 Subject: [PATCH 1/3] doc for creating custom cache storage backend. --- docs/topics/downloader-middleware.rst | 53 +++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0d976077b..f913b059d 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -496,6 +496,59 @@ In order to use this storage backend: .. _LevelDB: https://github.com/google/leveldb .. _leveldb python bindings: https://pypi.python.org/pypi/leveldb +.. _httpcache-storage-custom: + +Implementing custom cache storage backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can implement custom cache storage backend by creating a Python class that +defines the methods described below. + +.. module:: scrapy.extensions.httpcache + +.. class:: CacheStorage + + .. method:: open_spider(spider) + + This method gets called after a spider has been opened for crawling. + + :param spider: the spider which has been opened + :type spider: :class:`~scrapy.spiders.Spider` object + + .. method:: close_spider(spider) + + This method gets called after a spider has been closed. + + :param spider: the spider which has been closed + :type spider: :class:`~scrapy.spiders.Spider` object + + .. method:: retrieve_response(spider, request) + + Returns response if present in cache, or ``None`` otherwise. + + :param spider: the spider which generated the request + :type spider: :class:`~scrapy.spiders.Spider` object + + :param request: the request to find cached reponse for + :type request: :class:`~scrapy.http.Request` object + + .. method:: store_response(spider, request, response) + + Stores the given response in the cache. + + :param spider: the spider for which the response is intended + :type spider: :class:`~scrapy.spiders.Spider` object + + :param request: corresponding request the spider generated + :type request: :class:`~scrapy.http.Request` object + + :param response: the response to store in the cache + :type response: :class:`~scrapy.http.Response` object + +In order to use your storage backend, set: + +* :setting:`HTTPCACHE_STORAGE` to path of your custom storage class. + HTTPCache middleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 9c9bca4e1c7984089c44f3a44e7594e06307b12f Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Wed, 27 Mar 2019 18:29:48 +0530 Subject: [PATCH 2/3] make suggested changes. --- docs/topics/downloader-middleware.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index f913b059d..dfbcdb8fa 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -349,7 +349,7 @@ HttpCacheMiddleware * :ref:`httpcache-storage-leveldb` You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE` - setting. Or you can also implement your own storage backend. + setting. Or you can also :ref:`implement your own storage backend. ` Scrapy ships with two HTTP cache policies: @@ -498,10 +498,10 @@ In order to use this storage backend: .. _httpcache-storage-custom: -Implementing custom cache storage backend -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Writing your own storage backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You can implement custom cache storage backend by creating a Python class that +You can implement a cache storage backend by creating a Python class that defines the methods described below. .. module:: scrapy.extensions.httpcache @@ -510,14 +510,16 @@ defines the methods described below. .. method:: open_spider(spider) - This method gets called after a spider has been opened for crawling. + This method gets called after a spider has been opened for crawling. It handles + the :signal:`open_spider ` signal. :param spider: the spider which has been opened :type spider: :class:`~scrapy.spiders.Spider` object .. method:: close_spider(spider) - This method gets called after a spider has been closed. + This method gets called after a spider has been closed. It handles + the :signal:`close_spider ` signal. :param spider: the spider which has been closed :type spider: :class:`~scrapy.spiders.Spider` object @@ -539,7 +541,7 @@ defines the methods described below. :param spider: the spider for which the response is intended :type spider: :class:`~scrapy.spiders.Spider` object - :param request: corresponding request the spider generated + :param request: the corresponding request the spider generated :type request: :class:`~scrapy.http.Request` object :param response: the response to store in the cache @@ -547,7 +549,7 @@ defines the methods described below. In order to use your storage backend, set: -* :setting:`HTTPCACHE_STORAGE` to path of your custom storage class. +* :setting:`HTTPCACHE_STORAGE` to the Python import path of your custom storage class. HTTPCache middleware settings From da82ede8a0751cf3e8496f35252eb0dcef4f197e Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Sat, 25 May 2019 17:19:10 +0530 Subject: [PATCH 3/3] describe method as a command --- docs/topics/downloader-middleware.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index dfbcdb8fa..236150059 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -526,7 +526,7 @@ defines the methods described below. .. method:: retrieve_response(spider, request) - Returns response if present in cache, or ``None`` otherwise. + Return response if present in cache, or ``None`` otherwise. :param spider: the spider which generated the request :type spider: :class:`~scrapy.spiders.Spider` object @@ -536,7 +536,7 @@ defines the methods described below. .. method:: store_response(spider, request, response) - Stores the given response in the cache. + Store the given response in the cache. :param spider: the spider for which the response is intended :type spider: :class:`~scrapy.spiders.Spider` object