8.0 KiB
Spider Middleware
The spider middleware is a framework of hooks into Scrapy's spider processing mechanism where you can plug custom functionality to process the requests that are sent to :ref:`topics-spiders` for processing and to process the responses and item that are generated from spiders.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Activating a spider middleware
To activate a spider middleware component, add it to the :setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the middleware class path and their values are the middleware orders.
System Message: ERROR/3 (<stdin>, line 17); backlink
Unknown interpreted text role "setting".Here's an example:
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
}
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the :setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled middlewares: the first middleware is the one closer to the engine and the last is the one closer to the spider.
System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "setting".To decide which order to assign to your middleware see the :setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied.
System Message: ERROR/3 (<stdin>, line 33); backlink
Unknown interpreted text role "setting".If you want to disable a builtin middleware (the ones defined in :setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it in your project :setting:`SPIDER_MIDDLEWARES` setting and assign None as its value. For example, if you want to disable the off-site middleware:
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': None,
}
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
Writing your own spider middleware
Writing your own spider middleware is easy. Each middleware component is a single Python class that defines one or more of the following methods:
System Message: ERROR/3 (<stdin>, line 59)
Unknown directive type "method".
.. method:: process_spider_input(response, spider)
response is a :class:`~scrapy.http.Response` object spider is a :class:`~scrapy.spider.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "class".This method is called for each request that goes through the spider middleware.
process_spider_input() should return either None or an iterable of :class:`~scrapy.http.Response` or :class:`~scrapy.http.Item` objects.
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "class".If returns None, Scrapy will continue processing this response, executing all other middlewares until, finally, the response is handled to the spider for processing.
If returns an iterable, Scrapy won't bother calling ANY other spider middleware process_spider_input() and will return the iterable back in the other direction for the process_spider_exception() and process_spider_output() methods to hook it.
System Message: ERROR/3 (<stdin>, line 77)
Unknown directive type "method".
.. method:: process_spider_output(response, result, spider)
response is a :class:`~scrapy.http.Response` object result is an iterable of :class:`~scrapy.http.Request` or :class:`~scrapy.item.Item` objects spider is a :class:`~scrapy.item.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 79); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 79); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 79); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 79); backlink
Unknown interpreted text role "class".This method is called with the results that are returned from the Spider, after it has processed the response.
process_spider_output() must return an iterable of :class:`~scrapy.http.Request` or :class:`~scrapy.item.Item` objects.
System Message: ERROR/3 (<stdin>, line 86); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 86); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 89)
Unknown directive type "method".
.. method:: process_spider_exception(request, exception, spider)
request is a :class:`~scrapy.http.Request` object. exception is an Exception object spider is a BaseSpider object
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "class".Scrapy calls process_spider_exception() when a spider or process_spider_input() (from a spider middleware) raises an exception.
process_spider_exception() should return either None or an iterable of :class:`~scrapy.http.Response` or :class:`~scrapy.item.Item` objects.
System Message: ERROR/3 (<stdin>, line 98); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 98); backlink
Unknown interpreted text role "class".If it returns None, Scrapy will continue processing this exception, executing any other process_spider_exception() in the middleware pipeline, until no middleware is left and the default exception handling kicks in.
If it returns an iterable the process_spider_output() pipeline kicks in, and no other process_spider_exception() will be called.
Built-in spider middleware reference
This page describes all spider middleware components that come with Scrapy. For information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide <topics-spider-middleware>`.
System Message: ERROR/3 (<stdin>, line 114); backlink
Unknown interpreted text role "ref".For a list of the components enabled by default (and their orders) see the :setting:`SPIDER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 118); backlink
Unknown interpreted text role "setting".DepthMiddleware
System Message: ERROR/3 (<stdin>, line 124)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.depth
DepthMiddleware is a scrape middleware used for tracking the depth of each Request inside the site being scraped. It can be used to limit the maximum depth to scrape or things like that.
The :class:`DepthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 132); backlink
Unknown interpreted text role "class".
:setting:`DEPTH_LIMIT` - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed.
System Message: ERROR/3 (<stdin>, line 135); backlink
Unknown interpreted text role "setting".
:setting:`DEPTH_STATS` - Whether to collect depth stats.
System Message: ERROR/3 (<stdin>, line 137); backlink
Unknown interpreted text role "setting".
HttpErrorMiddleware
System Message: ERROR/3 (<stdin>, line 142)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.httperror
Filter out response outside of a range of valid status codes.
This middleware filters out every response with status outside of the range 200<=status<300. Spiders can add more exceptions using handle_httpstatus_list spider attribute.
OffsiteMiddleware
System Message: ERROR/3 (<stdin>, line 155)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.offsite
Filters out Requests for URLs outside the domains covered by the spider.
This middleware filters out every request whose host names doesn't match :attr:`~scrapy.spider.BaseSpider.domain_name`, or the spider :attr:`~scrapy.spider.BaseSpider.domain_name` prefixed by "www.". Spider can add more domains to exclude using :attr:`~scrapy.spider.BaseSpider.extra_domain_names` attribute.
System Message: ERROR/3 (<stdin>, line 161); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 161); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 161); backlink
Unknown interpreted text role "attr".RequestLimitMiddleware
System Message: ERROR/3 (<stdin>, line 170)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.requestlimit
Limits the maximum number of requests in the scheduler for each spider. When a spider tries to schedule more than the allowed amount of requests, the new requests (returned by the spider) will be dropped.
The :class:`RequestLimitMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 178); backlink
Unknown interpreted text role "class".
:setting:`REQUESTS_QUEUE_SIZE` - If non zero, it will be used as an upper limit for the amount of requests that can be scheduled per domain. Can be set per spider using requests_queue_size attribute.
System Message: ERROR/3 (<stdin>, line 181); backlink
Unknown interpreted text role "setting".
RestrictMiddleware
System Message: ERROR/3 (<stdin>, line 188)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.restrict
Restricts crawling to fixed set of particular URLs.
The :class:`RestrictMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 194); backlink
Unknown interpreted text role "class".
:setting:`RESTRICT_TO_URLS` - Set of URLs allowed to crawl.
System Message: ERROR/3 (<stdin>, line 197); backlink
Unknown interpreted text role "setting".
UrlFilterMiddleware
System Message: ERROR/3 (<stdin>, line 202)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urlfilter
Canonicalizes URLs to filter out duplicated ones
UrlLengthMiddleware
System Message: ERROR/3 (<stdin>, line 211)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urllength
Filters out requests with URLs longer than URLLENGTH_LIMIT
The :class:`UrlLengthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 217); backlink
Unknown interpreted text role "class".
:setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
System Message: ERROR/3 (<stdin>, line 220); backlink
Unknown interpreted text role "setting".