RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Mention how to disable request filtering in documentation of DUPEFILTER_CLASS setting

This commit is contained in:
Tiago Cardoso 2017-04-22 14:36:44 +01:00
parent 3dee913d44
commit 00ee9eaeaf
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/topics/settings.rst
View File

@ -646,6 +646,13 @@ override its ``request_fingerprint`` method. This method should accept
scrapy :class:`~scrapy.http.Request` object and return its fingerprint
(a string).
You can disable filtering of duplicate requests by setting
:setting:`DUPEFILTER_CLASS` to ``'scrapy.dupefilters.BaseDupeFilter'``.
Be very careful about this however, because you can get into crawling loops.
It's usually a better idea to set the ``dont_filter`` parameter to
``True`` on the specific :class:`~scrapy.http.Request` that should not be
filtered.
.. setting:: DUPEFILTER_DEBUG
DUPEFILTER_DEBUG