3.7 KiB
Built-in spider middleware reference
This page describes all spider middleware components that come with Scrapy. For information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide <topics-spider-middleware>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".For a list of the components enabled by default (and their orders) see the :setting:`SPIDER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 11); backlink
Unknown interpreted text role "setting".Available spider middlewares
DepthMiddleware
System Message: ERROR/3 (<stdin>, line 20)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.depth
DepthMiddleware is a scrape middleware used for tracking the depth of each Request inside the site being scraped. It can be used to limit the maximum depth to scrape or things like that.
The :class:`DepthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 28); backlink
Unknown interpreted text role "class".
:setting:`DEPTH_LIMIT` - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed.
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "setting".
:setting:`DEPTH_STATS` - Whether to collect depth stats.
System Message: ERROR/3 (<stdin>, line 33); backlink
Unknown interpreted text role "setting".
HttpErrorMiddleware
System Message: ERROR/3 (<stdin>, line 38)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.httperror
Filter out response outside of a range of valid status codes.
This middleware filters out every response with status outside of the range 200<=status<300. Spiders can add more exceptions using handle_httpstatus_list spider attribute.
OffsiteMiddleware
System Message: ERROR/3 (<stdin>, line 51)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.offsite
Filters out Requests for URLs outside the domains covered by the spider.
This middleware filters out every request whose host names doesn't match :attr:`~scrapy.spider.BaseSpider.domain_name`, or the spider :attr:`~scrapy.spider.BaseSpider.domain_name` prefixed by "www.". Spider can add more domains to exclude using :attr:`~scrapy.spider.BaseSpider.extra_domain_names` attribute.
System Message: ERROR/3 (<stdin>, line 57); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 57); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 57); backlink
Unknown interpreted text role "attr".RequestLimitMiddleware
System Message: ERROR/3 (<stdin>, line 66)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.requestlimit
Limits the maximum number of requests in the scheduler for each spider. When a spider tries to schedule more than the allowed amount of requests, the new requests (returned by the spider) will be dropped.
The :class:`RequestLimitMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 74); backlink
Unknown interpreted text role "class".
:setting:`REQUESTS_QUEUE_SIZE` - If non zero, it will be used as an upper limit for the amount of requests that can be scheduled per domain. Can be set per spider using requests_queue_size attribute.
System Message: ERROR/3 (<stdin>, line 77); backlink
Unknown interpreted text role "setting".
RestrictMiddleware
System Message: ERROR/3 (<stdin>, line 84)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.restrict
Restricts crawling to fixed set of particular URLs.
The :class:`RestrictMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 90); backlink
Unknown interpreted text role "class".
:setting:`RESTRICT_TO_URLS` - Set of URLs allowed to crawl.
System Message: ERROR/3 (<stdin>, line 93); backlink
Unknown interpreted text role "setting".
UrlFilterMiddleware
System Message: ERROR/3 (<stdin>, line 98)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urlfilter
Canonicalizes URLs to filter out duplicated ones
UrlLengthMiddleware
System Message: ERROR/3 (<stdin>, line 107)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urllength
Filters out requests with URLs longer than URLLENGTH_LIMIT
The :class:`UrlLengthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 113); backlink
Unknown interpreted text role "class".
:setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
System Message: ERROR/3 (<stdin>, line 116); backlink
Unknown interpreted text role "setting".