mirror of https://github.com/scrapy/scrapy.git
866 B
866 B
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
Built-in scheduler middleware reference
This page describes all scheduler middleware components that come with Scrapy.
For a list of the components enabled by default (and their orders) see the :setting:`SCHEDULER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 10); backlink
Unknown interpreted text role "setting".Available scheduler middlewares
DuplicatesFilterMiddleware
System Message: ERROR/3 (<stdin>, line 19)
Unknown directive type "module".
.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter
Filter out already visited urls.
The :class:`DuplicatesFilterMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "class".
:setting:`DUPEFILTER_CLASS` - The class used to detect and filter duplicate requests.
System Message: ERROR/3 (<stdin>, line 28); backlink
Unknown interpreted text role "setting".