From 96c3cdbec24e010aaa43a9d83ee6226ebdad557d Mon Sep 17 00:00:00 2001 From: Daniel Grana Date: Fri, 24 Jul 2009 12:59:38 -0300 Subject: [PATCH] improve OffsiteMiddleware reference docs --HG-- extra : rebase_source : 3ed3f23fc1ec63b521ead029c5749898f3ab05d7 --- docs/ref/spider-middleware.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/ref/spider-middleware.rst b/docs/ref/spider-middleware.rst index e2497b953..ca90a28fb 100644 --- a/docs/ref/spider-middleware.rst +++ b/docs/ref/spider-middleware.rst @@ -54,6 +54,12 @@ OffsiteMiddleware Filters out Requests for URLs outside the domains covered by the spider. + This middleware filters out every request whose host names doesn't match + :attr:`~scrapy.spider.BaseSpider.domain_name`, or the spider + :attr:`~scrapy.spider.BaseSpider.domain_name` prefixed by "www.". + Spider can add more domains to exclude using + :attr:`~scrapy.spider.BaseSpider.extra_domain_names` attribute. + RequestLimitMiddleware ----------------------