RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/scheduler-middleware.rst

919 B
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scheduler middleware

Built-in scheduler middleware reference

This page describes all scheduler middleware components that come with Scrapy.

For a list of the components enabled by default (and their orders) see the :setting:`SCHEDULER_MIDDLEWARES_BASE` setting.

System Message: ERROR/3 (<stdin>, line 16); backlink

Unknown interpreted text role "setting".

DuplicatesFilterMiddleware

System Message: ERROR/3 (<stdin>, line 22)

Unknown directive type "module".

.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter
   :synopsis: Duplicates Filter Scheduler Middleware

Filter out already visited urls.

The :class:`DuplicatesFilterMiddleware` can be configured through the following settings (see the settings documentation for more info):

System Message: ERROR/3 (<stdin>, line 29); backlink

Unknown interpreted text role "class".

  • :setting:`DUPEFILTER_CLASS` - The class used to detect and filter duplicate requests.

    System Message: ERROR/3 (<stdin>, line 32); backlink

    Unknown interpreted text role "setting".

</html>