From 3998a0cb5809b2784e27a89c6d6ae3e33a224d70 Mon Sep 17 00:00:00 2001 From: Ismael Carnales Date: Fri, 11 Sep 2009 11:58:53 -0300 Subject: [PATCH] added more scheduler middleware documentation, and moved it to experimental --HG-- rename : docs/topics/scheduler-middleware.rst => docs/experimental/scheduler-middleware.rst --- docs/experimental/index.rst | 1 + docs/experimental/scheduler-middleware.rst | 106 +++++++++++++++++++++ docs/index.rst | 1 - docs/topics/scheduler-middleware.rst | 34 ------- 4 files changed, 107 insertions(+), 35 deletions(-) create mode 100644 docs/experimental/scheduler-middleware.rst delete mode 100644 docs/topics/scheduler-middleware.rst diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst index a6546ba36..fd841767c 100644 --- a/docs/experimental/index.rst +++ b/docs/experimental/index.rst @@ -21,3 +21,4 @@ it's properly merged) . Use at your own risk. images djangoitems + scheduler-middleware diff --git a/docs/experimental/scheduler-middleware.rst b/docs/experimental/scheduler-middleware.rst new file mode 100644 index 000000000..7dae54dc3 --- /dev/null +++ b/docs/experimental/scheduler-middleware.rst @@ -0,0 +1,106 @@ +.. _topics-scheduler-middleware: + +==================== +Scheduler middleware +==================== + +The scheduler middleware is a framework of hooks in the Scrapy's scheduling +mechanism where you can plug custom functionality to process request being +enqueued. + +Activating a scheduler middleware +================================= + +To activate a scheduler middleware component, add it to the +:setting:`SCHEDULER_MIDDLEWARES` setting, which is a dict whose keys are the +middleware class path and their values are the middleware orders. + +Here's an example:: + + SCHEDULER_MIDDLEWARES = { + 'myproject.middlewares.CustomSchedulerMiddleware': 543, + } + +The :setting:`SCHEDULER_MIDDLEWARES` setting is merged with the +:setting:`SCHEDULER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant +to be overridden) and then sorted by order to get the final sorted list of +enabled middlewares: the first middleware is the one closer to the engine and +the last is the one closer to the spider. + +To decide which order to assign to your middleware see the +:setting:`SCHEDULER_MIDDLEWARES_BASE` setting and pick a value according to +where you want to insert the middleware. The order does matter because each +middleware performs a different action and your middleware could depend on some +previous (or subsequent) middleware being applied. + +If you want to disable a builtin middleware (the ones defined in +:setting:`SCHEDULER_MIDDLEWARES_BASE`, and enabled by default) you must define it +in your project :setting:`SCHEDULER_MIDDLEWARES` setting and assign `None` as its +value. For example, if you want to disable the duplicates filter middleware:: + + SPIDER_MIDDLEWARES = { + 'myproject.middlewares.CustomSchedulerMiddleware': 543, + 'scrapy.contrib.spidermiddleware.duplicatesfilter.DuplicatesFilterMiddleware: None, + } + +Finally, keep in mind that some middlewares may need to be enabled through a +particular setting. See each middleware documentation for more info. + +Writing your own scheduler middleware +===================================== + +Writing your own scheduler middleware is easy. Each middleware component is a +single Python class that defines one or more of the following methods: + +.. module:: scrapy.contrib.schedulermiddleware + +.. class:: SchedulerMiddleware + + .. method:: enqueue_request(domain, request) + + :meth:`enqueue_request` should return either ``None``, a + :class:`~scrapy.http.Response` object or a ``Deferred``. + + :param domain: the domain originating the request + :type domain: string + + :param requests: the request to be enqueued + :type request: :class:`~scrapy.http.Request` object + + .. method:: open_domain(domain) + + :param domain: the domain being opened + :type domain: string + + .. method:: close_domain(domain) + + :param domain: the domain being closed + :type domain: string + +.. _topics-scheduler-middleware-ref: + +Built-in scheduler middleware reference +======================================== + +This page describes all scheduler middleware components that come with +Scrapy. + +For a list of the components enabled by default (and their orders) see the +:setting:`SCHEDULER_MIDDLEWARES_BASE` setting. + +DuplicatesFilterMiddleware +-------------------------- + +.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter + :synopsis: Duplicates Filter Scheduler Middleware + +.. class:: DuplicatesFilterMiddleware + + Filter out already visited urls. + + The :class:`DuplicatesFilterMiddleware` can be configured through the following + settings (see the settings documentation for more info): + + * :setting:`DUPEFILTER_CLASS` - The class used to detect and filter + duplicate requests. + diff --git a/docs/index.rst b/docs/index.rst index 906ba38ab..497040d40 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -138,7 +138,6 @@ Extending Scrapy topics/architecture topics/downloader-middleware topics/spider-middleware - topics/scheduler-middleware topics/extensions :doc:`topics/architecture` diff --git a/docs/topics/scheduler-middleware.rst b/docs/topics/scheduler-middleware.rst deleted file mode 100644 index f20aaa645..000000000 --- a/docs/topics/scheduler-middleware.rst +++ /dev/null @@ -1,34 +0,0 @@ -.. _topics-scheduler-middleware: - -==================== -Scheduler middleware -==================== - - -.. _topics-scheduler-middleware-ref: - -Built-in scheduler middleware reference -======================================== - -This page describes all scheduler middleware components that come with -Scrapy. - -For a list of the components enabled by default (and their orders) see the -:setting:`SCHEDULER_MIDDLEWARES_BASE` setting. - -DuplicatesFilterMiddleware --------------------------- - -.. module:: scrapy.contrib.schedulermiddleware.duplicatesfilter - :synopsis: Duplicates Filter Scheduler Middleware - -.. class:: DuplicatesFilterMiddleware - - Filter out already visited urls. - - The :class:`DuplicatesFilterMiddleware` can be configured through the following - settings (see the settings documentation for more info): - - * :setting:`DUPEFILTER_CLASS` - The class used to detect and filter - duplicate requests. -