From c67a232dfe73e4630b16afef3dac0a6c9e95e8ce Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Tue, 9 Apr 2019 22:33:02 +0530 Subject: [PATCH 1/2] docs for writing custom policy --- docs/topics/downloader-middleware.rst | 60 ++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0d976077b..5f9f09a45 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -357,7 +357,7 @@ HttpCacheMiddleware * :ref:`httpcache-policy-dummy` You can change the HTTP cache policy with the :setting:`HTTPCACHE_POLICY` - setting. Or you can also implement your own policy. + setting. Or you can also :ref:`implement your own policy. ` .. reqmeta:: dont_cache @@ -429,6 +429,64 @@ In order to use this policy, set: * :setting:`HTTPCACHE_POLICY` to ``scrapy.extensions.httpcache.RFC2616Policy`` +.. _httpcache-policy-custom: + +Writing your own policy +~~~~~~~~~~~~~~~~~~~~~~~ + +You can implement a cache policy by creating a Python class that +defines the methods described below. + +.. module:: scrapy.extensions.httpcache + +.. class:: CachePolicy + + .. method:: should_cache_request(request) + + Returns whether the request is cacheable or not + + :param request: the request to check + :type request: :class:`~scrapy.http.Request` object + + .. method:: should_cache_response(response, request) + + Returns whether to cache response or not + + :param cachedresponse: the response to check + :type cachedresponse: :class:`~scrapy.http.Response` object + + :param request: the corresponding request + :type request: :class:`~scrapy.http.Request` object + + .. method:: is_cached_response_fresh(cachedresponse, request) + + Returns whether cached response has expired or not + + :param cachedresponse: the cached response to check for freshness + :type cachedresponse: :class:`~scrapy.http.Response` object + + :param request: the request for which the cached response is intended + :type request: :class:`~scrapy.http.Request` object + + .. method:: is_cached_response_valid(cachedresponse, response, request) + + Returns whether cached response is valid or not + + :param cachedresponse: the cached response to validate + :type cachedresponse: :class:`~scrapy.http.Response` object + + :param response: the response from the server to use for validation + :type response: :class:`~scrapy.http.Response` object + + :param request: the request for which the cached response is intended + :type request: :class:`~scrapy.http.Request` object + + +In order to use this policy, set: + +* :setting:`HTTPCACHE_POLICY` to the Python import path of your custom policy class. + + .. _httpcache-storage-fs: Filesystem storage backend (default) From 7538718464b0e3b48707b919cdb3f8bce2f1ebe5 Mon Sep 17 00:00:00 2001 From: Anubhav Patel Date: Mon, 29 Apr 2019 22:02:08 +0530 Subject: [PATCH 2/2] makes suggested changes --- docs/topics/downloader-middleware.rst | 36 +++++---------------------- 1 file changed, 6 insertions(+), 30 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 5f9f09a45..6b1eb5a45 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -357,7 +357,7 @@ HttpCacheMiddleware * :ref:`httpcache-policy-dummy` You can change the HTTP cache policy with the :setting:`HTTPCACHE_POLICY` - setting. Or you can also :ref:`implement your own policy. ` + setting. Or you can also :ref:`implement your own policy `. .. reqmeta:: dont_cache @@ -434,7 +434,7 @@ In order to use this policy, set: Writing your own policy ~~~~~~~~~~~~~~~~~~~~~~~ -You can implement a cache policy by creating a Python class that +You can implement a cache policy by creating a class that defines the methods described below. .. module:: scrapy.extensions.httpcache @@ -443,43 +443,19 @@ defines the methods described below. .. method:: should_cache_request(request) - Returns whether the request is cacheable or not - - :param request: the request to check - :type request: :class:`~scrapy.http.Request` object + Return True if the :class:`~scrapy.http.Request` is cacheable otherwise False .. method:: should_cache_response(response, request) - Returns whether to cache response or not - - :param cachedresponse: the response to check - :type cachedresponse: :class:`~scrapy.http.Response` object - - :param request: the corresponding request - :type request: :class:`~scrapy.http.Request` object + Return True if :class:`~scrapy.http.Response` should be cached otherwise False .. method:: is_cached_response_fresh(cachedresponse, request) - Returns whether cached response has expired or not - - :param cachedresponse: the cached response to check for freshness - :type cachedresponse: :class:`~scrapy.http.Response` object - - :param request: the request for which the cached response is intended - :type request: :class:`~scrapy.http.Request` object + Return True if cached :class:`~scrapy.http.Response` has expired otherwise False .. method:: is_cached_response_valid(cachedresponse, response, request) - Returns whether cached response is valid or not - - :param cachedresponse: the cached response to validate - :type cachedresponse: :class:`~scrapy.http.Response` object - - :param response: the response from the server to use for validation - :type response: :class:`~scrapy.http.Response` object - - :param request: the request for which the cached response is intended - :type request: :class:`~scrapy.http.Request` object + Return True if cached :class:`~scrapy.http.Response` is valid otherwise False In order to use this policy, set: