From 00ee9eaeafcd3b3af43132c4513754627f97a886 Mon Sep 17 00:00:00 2001 From: Tiago Cardoso Date: Sat, 22 Apr 2017 14:36:44 +0100 Subject: [PATCH] Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting --- docs/topics/settings.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 8367b1092..9bf07588b 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -646,6 +646,13 @@ override its ``request_fingerprint`` method. This method should accept scrapy :class:`~scrapy.http.Request` object and return its fingerprint (a string). +You can disable filtering of duplicate requests by setting +:setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``. +Be very careful about this however, because you can get into crawling loops. +It's usually a better idea to set the ``dont_filter`` parameter to +``True`` on the specific :class:`~scrapy.http.Request` that should not be +filtered. + .. setting:: DUPEFILTER_DEBUG DUPEFILTER_DEBUG