From b1c0280616ffd3333acc6408ea1287392866f1de Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Tue, 10 Aug 2010 16:59:49 -0300 Subject: [PATCH] removed scheduler middleware doc, as scheduler middleware will be removed soon --- docs/experimental/index.rst | 1 - docs/experimental/scheduler-middleware.rst | 99 ---------------------- 2 files changed, 100 deletions(-) delete mode 100644 docs/experimental/scheduler-middleware.rst diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst index f63ceddd9..f159e8c51 100644 --- a/docs/experimental/index.rst +++ b/docs/experimental/index.rst @@ -20,5 +20,4 @@ it's properly merged) . Use at your own risk. :maxdepth: 1 djangoitems - scheduler-middleware crawlspider-v2 diff --git a/docs/experimental/scheduler-middleware.rst b/docs/experimental/scheduler-middleware.rst deleted file mode 100644 index 8fb6358a2..000000000 --- a/docs/experimental/scheduler-middleware.rst +++ /dev/null @@ -1,99 +0,0 @@ -.. _topics-scheduler-middleware: - -==================== -Scheduler middleware -==================== - -The scheduler middleware is a framework of hooks in the Scrapy scheduling -mechanism where you can plug custom functionality to process requests being -enqueued. - -Activating a scheduler middleware -================================= - -To activate a scheduler middleware component, add it to the -:setting:`SCHEDULER_MIDDLEWARES` setting, which is a dict whose keys are the -middleware class path and their values are the middleware orders. - -Here's an example:: - - SCHEDULER_MIDDLEWARES = { - 'myproject.middlewares.CustomSchedulerMiddleware': 543, - } - -The :setting:`SCHEDULER_MIDDLEWARES` setting is merged with the -:setting:`SCHEDULER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant -to be overridden) and then sorted by order to get the final sorted list of -enabled middlewares: the first middleware is the one closer to the engine and -the last is the one closer to the scheduler. - -To decide which order to assign to your middleware see the -:setting:`SCHEDULER_MIDDLEWARES_BASE` setting and pick a value according to -where you want to insert the middleware. The order does matter because each -middleware performs a different action and your middleware could depend on some -previous (or subsequent) middleware being applied. - -If you want to disable a builtin middleware (the ones defined in -:setting:`SCHEDULER_MIDDLEWARES_BASE`, and enabled by default) you must define it -in your project :setting:`SCHEDULER_MIDDLEWARES` setting and assign `None` as its -value. For example, if you want to disable the duplicates filter middleware:: - - SCHEDULER_MIDDLEWARES = { - 'myproject.middlewares.CustomSchedulerMiddleware': 543, - 'scrapy.contrib.schedulermiddleware.duplicatesfilter.DuplicatesFilterMiddleware: None, - } - -Finally, keep in mind that some middlewares may need to be enabled through a -particular setting. See each middleware documentation for more info. - -Writing your own scheduler middleware -===================================== - -Writing your own scheduler middleware is easy. Each middleware component is a -single Python class that defines one or more of the following methods: - -.. module:: scrapy.contrib.schedulermiddleware - -.. class:: SchedulerMiddleware - - .. method:: enqueue_request(spider, request) - - Process the given request which is being enqueued. This method can return - None to avoid the request from being scheduled. - - :meth:`enqueue_request` should return either ``None``, a - :class:`~scrapy.http.Response` object or a ``Deferred``. - - :param spider: the spider originating the request - :type spider: :class:`~scrapy.spider.BaseSpider` object - - :param requests: the request to be enqueued - :type request: :class:`~scrapy.http.Request` object - -.. _topics-scheduler-middleware-ref: - -Built-in scheduler middleware reference -======================================== - -This page describes all scheduler middleware components that come with -Scrapy. - -For a list of the components enabled by default (and their orders) see the -:setting:`SCHEDULER_MIDDLEWARES_BASE` setting. - -DuplicatesFilterMiddleware --------------------------- - -.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter - :synopsis: Duplicates Filter Scheduler Middleware - -.. class:: DuplicatesFilterMiddleware - - Filter out already visited urls. - - The :class:`DuplicatesFilterMiddleware` can be configured through the following - settings (see the settings documentation for more info): - - * :setting:`DUPEFILTER_CLASS` - The class that implements the duplicate - filtering policy. -