10 KiB
Spider Middleware
The spider middleware is a framework of hooks into Scrapy's spider processing mechanism where you can plug custom functionality to process the requests that are sent to :ref:`topics-spiders` for processing and to process the responses and items that are generated from spiders.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Activating a spider middleware
To activate a spider middleware component, add it to the :setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the middleware class path and their values are the middleware orders.
System Message: ERROR/3 (<stdin>, line 17); backlink
Unknown interpreted text role "setting".Here's an example:
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
}
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the :setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled middlewares: the first middleware is the one closer to the engine and the last is the one closer to the spider.
System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "setting".To decide which order to assign to your middleware see the :setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied.
System Message: ERROR/3 (<stdin>, line 33); backlink
Unknown interpreted text role "setting".If you want to disable a builtin middleware (the ones defined in :setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it in your project :setting:`SPIDER_MIDDLEWARES` setting and assign None as its value. For example, if you want to disable the off-site middleware:
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': None,
}
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
Writing your own spider middleware
Writing your own spider middleware is easy. Each middleware component is a single Python class that defines one or more of the following methods:
System Message: ERROR/3 (<stdin>, line 58)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware
System Message: ERROR/3 (<stdin>, line 62)
Unknown directive type "method".
.. method:: process_spider_input(response, spider)
This method is called for each response that goes through the spider
middleware and into the spider, for processing.
:meth:`process_spider_input` should return ``None`` or raise and
exception.
If it returns ``None``, Scrapy will continue processing this response,
executing all other middlewares until, finally, the response is handled
to the spider for processing.
If it raises an exception, Scrapy won't bother calling any other spider
middleware :meth:`process_spider_input` and will call the request
errback. The output of the errback is chained back in the other
direction for :meth:`process_spider_output` to process it, or
:meth:`process_spider_exception` if it raised an exception.
:param reponse: the response being processed
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider for which this response is intended
:type spider: :class:`~scrapy.spider.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 87)
Unknown directive type "method".
.. method:: process_spider_output(response, result, spider)
This method is called with the results returned from the Spider, after
it has processed the response.
:meth:`process_spider_output` must return an iterable of
:class:`~scrapy.http.Request` or :class:`~scrapy.item.Item` objects.
:param response: the response which generated this output from the
spider
:type response: class:`~scrapy.http.Response` object
:param result: the result returned by the spider
:type result: an iterable of :class:`~scrapy.http.Request` or
:class:`~scrapy.item.Item` objects
:param spider: the spider whose result is being processed
:type spider: :class:`~scrapy.item.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 107)
Unknown directive type "method".
.. method:: process_spider_exception(response, exception, spider)
This method is called when when a spider or :meth:process_spider_input:
method (from other spider middleware) raises an exception.
:meth:`process_spider_exception` should return either ``None`` or an
iterable of :class:`~scrapy.http.Response` or
:class:`~scrapy.item.Item` objects.
If it returns ``None``, Scrapy will continue processing this exception,
executing any other :meth:`process_spider_exception` in the following
middleware components, until no middleware components are left and the
exception reaches the engine (where it's logged and discarded).
If it returns an iterable the :meth:`process_spider_output` pipeline
kicks in, and no other :meth:`process_spider_exception` will be called.
:param response: the response being processed when the exception was
raised
:type response: :class:`~scrapy.http.Response` object
:param exception: the exception raised
:type exception: `Exception`_ object
:param spider: the spider which raised the exception
:type spider: :class:`scrapy.spider.BaseSpider` object
Built-in spider middleware reference
This page describes all spider middleware components that come with Scrapy. For information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide <topics-spider-middleware>`.
System Message: ERROR/3 (<stdin>, line 142); backlink
Unknown interpreted text role "ref".For a list of the components enabled by default (and their orders) see the :setting:`SPIDER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 146); backlink
Unknown interpreted text role "setting".DepthMiddleware
System Message: ERROR/3 (<stdin>, line 152)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.depth :synopsis: Depth Spider Middleware
DepthMiddleware is a scrape middleware used for tracking the depth of each Request inside the site being scraped. It can be used to limit the maximum depth to scrape or things like that.
The :class:`DepthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 161); backlink
Unknown interpreted text role "class".
:setting:`DEPTH_LIMIT` - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed.
System Message: ERROR/3 (<stdin>, line 164); backlink
Unknown interpreted text role "setting".
:setting:`DEPTH_STATS` - Whether to collect depth stats.
System Message: ERROR/3 (<stdin>, line 166); backlink
Unknown interpreted text role "setting".
HttpErrorMiddleware
System Message: ERROR/3 (<stdin>, line 171)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.httperror :synopsis: HTTP Error Spider Middleware
Filter out unsuccessful (erroneous) HTTP responses so that spiders don't have to deal with them, which (most of the times) imposes an overhead, consumes more resources, and makes the spider logic more complex.
According to the HTTP standard, successful responses are those whose status codes are in the 200-300 range.
If you still want to process response codes outside that range, you can specify which response codes the spider is able to handle using the handle_httpstatus_list spider attribute.
For example, if you want your spider to handle 404 responses you can do this:
class MySpider(CrawlSpider): handle_httpstatus_list = [404]Keep in mind, however, that it's usually a bad idea to handle non-200 responses, unless you really know what you're doing.
For more information see: HTTP Status Code Definitions.
OffsiteMiddleware
System Message: ERROR/3 (<stdin>, line 205)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.offsite :synopsis: Offiste Spider Middleware
Filters out Requests for URLs outside the domains covered by the spider.
This middleware filters out every request whose host names aren't in the spider's :attr:`~scrapy.spider.BaseSpider.allowed_domains` attribute.
System Message: ERROR/3 (<stdin>, line 212); backlink
Unknown interpreted text role "attr".When your spider returns a request for a domain not belonging to those covered by the spider, this middleware will log a debug message similar to this one:
DEBUG: Filtered offsite request to 'www.othersite.com': <GET http://www.othersite.com/some/page.html>
To avoid filling the log with too much noise, it will only print one of these messages for each new domain filtered. So, for example, if another request for www.othersite.com is filtered, no log message will be printed. But if a request for someothersite.com is filtered, a message will be printed (but only for the first request filtred).
RefererMiddleware
System Message: ERROR/3 (<stdin>, line 230)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.referer :synopsis: Referer Spider Middleware
Populates Request referer field, based on the Response which originated it.
RequestLimitMiddleware
System Message: ERROR/3 (<stdin>, line 240)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.requestlimit :synopsis: Request limit Spider Middleware
Limits the maximum number of requests in the scheduler for each spider. When a spider tries to schedule more than the allowed amount of requests, the new requests (returned by the spider) will be dropped.
The :class:`RequestLimitMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 249); backlink
Unknown interpreted text role "class".
:setting:`REQUESTS_QUEUE_SIZE` - If non zero, it will be used as an upper limit for the amount of requests that can be scheduled per domain. Can be set per spider using requests_queue_size attribute.
System Message: ERROR/3 (<stdin>, line 252); backlink
Unknown interpreted text role "setting".
UrlFilterMiddleware
System Message: ERROR/3 (<stdin>, line 259)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urlfilter :synopsis: URL Filter Spider Middleware
Canonicalizes URLs to filter out duplicated ones
UrlLengthMiddleware
System Message: ERROR/3 (<stdin>, line 269)
Unknown directive type "module".
.. module:: scrapy.contrib.spidermiddleware.urllength :synopsis: URL Length Spider Middleware
Filters out requests with URLs longer than URLLENGTH_LIMIT
The :class:`UrlLengthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 276); backlink
Unknown interpreted text role "class".
:setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "setting".