Added Scheduler middleware reference documentation

docs/ref/index.rst

@@ -19,6 +19,7 @@ This section documents the Scrapy |version| API. For more information see :ref:`
    extension-manager
    extensions
    downloader-middleware
+   scheduler-middleware
    link-extractors
 
 * :ref:`topics-stats-api`

docs/ref/scheduler-middleware.rst

@@ -0,0 +1,30 @@
+.. _ref-scheduler-middleware:
+
+========================================
+Built-in scheduler middleware reference
+========================================
+
+This page describes all scheduler middleware components that come with
+Scrapy. 
+
+For a list of the components enabled by default (and their orders) see the
+:setting:`SCHEDULER_MIDDLEWARES_BASE` setting.
+
+Available downloader middlewares
+================================
+
+DuplicatesFilterMiddleware
+--------------------------
+
+.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter
+
+.. class:: DuplicatesFilterMiddleware
+
+   Filter out already visited urls.
+
+   The :class:`DuplicatesFilterMiddleware` can be configured through the following
+   settings (see the settings documentation for more info):
+
+      * :setting:`DUPEFILTER_CLASS` - The class used to detect and filter
+        duplicate requests.
+

scrapy/contrib/schedulermiddleware/duplicatesfilter.py

@@ -1,5 +1,7 @@
 """
 DuplicatesFilterMiddleware: Filter out already visited urls
+
+See documentation in docs/ref/scheduler-middleware.rst
 """
 
 from scrapy.core.exceptions import IgnoreRequest, NotConfigured
@@ -7,7 +9,7 @@ from scrapy.utils.misc import load_object
 from scrapy.conf import settings
 
 class DuplicatesFilterMiddleware(object):
-    """Filter out already seen requests to avoid visiting pages more than once."""
+
     def __init__(self):
         clspath = settings.get('DUPEFILTER_CLASS')
         if not clspath: