3.6 KiB
Spider Middleware
The spider middleware is a framework of hooks into Scrapy's spider processing mechanism where you can plug custom functionality to process the requests that are sent to :ref:`topics-spiders` for processing and to process the responses and item that are generated from spiders.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Activating a spider middleware
To activate a middleware component, add it to the :setting:`SPIDER_MIDDLEWARES` list in your Scrapy settings. In :setting:`SPIDER_MIDDLEWARES`, each middleware component is represented by a string: the full Python path to the middleware's class name. For example:
System Message: ERROR/3 (<stdin>, line 15); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 15); backlink
Unknown interpreted text role "setting".SPIDER_MIDDLEWARES = [
'scrapy.contrib.spidermiddleware.limit.RequestLimitMiddleware',
'scrapy.contrib.spidermiddleware.restrict.RestrictMiddleware',
'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware',
'scrapy.contrib.spidermiddleware.referer.RefererMiddleware',
'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware',
'scrapy.contrib.spidermiddleware.depth.DepthMiddleware',
]
The first (top) middleware is the one closer to the engine, the last (bottom) middleware is the one closer to the spider.
Writing your own spider middleware
Writing your own spider middleware is easy. Each middleware component is a single Python class that defines one or more of the following methods:
System Message: ERROR/3 (<stdin>, line 39)
Unknown directive type "method".
.. method:: process_spider_input(response, spider)
response is a :class:`~scrapy.http.Response` object spider is a :class:`~scrapy.spider.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 41); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 41); backlink
Unknown interpreted text role "class".This method is called for each request that goes through the spider middleware.
process_spider_input() should return either None or an iterable of :class:`~scrapy.http.Response` or :class:`~scrapy.http.ScrapedItem` objects.
System Message: ERROR/3 (<stdin>, line 46); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 46); backlink
Unknown interpreted text role "class".If returns None, Scrapy will continue processing this response, executing all other middlewares until, finally, the response is handled to the spider for processing.
If returns an iterable, Scrapy won't bother calling ANY other spider middleware process_spider_input() and will return the iterable back in the other direction for the process_spider_exception() and process_spider_output() methods to hook it.
System Message: ERROR/3 (<stdin>, line 57)
Unknown directive type "method".
.. method:: process_spider_output(response, result, spider)
response is a :class:`~scrapy.http.Response` object result is an iterable of :class:`~scrapy.http.Request` or :class:`~scrapy.item.ScrapedItem` objects spider is a :class:`~scrapy.item.BaseSpider` object
System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 59); backlink
Unknown interpreted text role "class".This method is called with the results that are returned from the Spider, after it has processed the response.
process_spider_output() must return an iterable of :class:`~scrapy.http.Request` or :class:`~scrapy.item.ScrapedItem` objects.
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 69)
Unknown directive type "method".
.. method:: process_spider_exception(request, exception, spider)
request is a :class:`~scrapy.http.Request` object. exception is an Exception object spider is a BaseSpider object
System Message: ERROR/3 (<stdin>, line 71); backlink
Unknown interpreted text role "class".Scrapy calls process_spider_exception() when a spider or process_spider_input() (from a spider middleware) raises an exception.
process_spider_exception() should return either None or an iterable of :class:`~scrapy.http.Response` or :class:`~scrapy.item.ScrapedItem` objects.
System Message: ERROR/3 (<stdin>, line 78); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 78); backlink
Unknown interpreted text role "class".If it returns None, Scrapy will continue processing this exception, executing any other process_spider_exception() in the middleware pipeline, until no middleware is left and the default exception handling kicks in.
If it returns an iterable the process_spider_output() pipeline kicks in, and no other process_spider_exception() will be called.