17 KiB
Spider Middleware
The spider middleware is a framework of hooks into Scrapy's spider processing mechanism where you can plug custom functionality to process the responses that are sent to :ref:`topics-spiders` for processing and to process the requests and items that are generated from spiders.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Activating a spider middleware
To activate a spider middleware component, add it to the :setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the middleware class path and their values are the middleware orders.
System Message: ERROR/3 (<stdin>, line 17); backlink
Unknown interpreted text role "setting".Here's an example:
System Message: WARNING/2 (<stdin>, line 23)
Cannot analyze code. Pygments package not found.
.. code-block:: python
SPIDER_MIDDLEWARES = {
"myproject.middlewares.CustomSpiderMiddleware": 543,
}
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the :setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled middlewares: the first middleware is the one closer to the engine and the last is the one closer to the spider. In other words, the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_input` method of each middleware will be invoked in increasing middleware order (100, 200, 300, ...), and the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method of each middleware will be invoked in decreasing order.
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "meth".To decide which order to assign to your middleware see the :setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied.
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "setting".If you want to disable a builtin middleware (the ones defined in :setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it in your project :setting:`SPIDER_MIDDLEWARES` setting and assign None as its value. For example, if you want to disable the off-site middleware:
System Message: ERROR/3 (<stdin>, line 46); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 46); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 51)
Cannot analyze code. Pygments package not found.
.. code-block:: python
SPIDER_MIDDLEWARES = {
"scrapy.spidermiddlewares.referer.RefererMiddleware": None,
"myproject.middlewares.CustomRefererSpiderMiddleware": 700,
}
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
Writing your own spider middleware
Each spider middleware is a :ref:`component <topics-components>` that defines one or more of these methods:
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 69)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares
System Message: ERROR/3 (<stdin>, line 73)
Unknown directive type "method".
.. method:: process_start(start: AsyncIterator[Any], /) -> AsyncIterator[Any]
:async:
Iterate over the output of :meth:`~scrapy.Spider.start` or that
of the :meth:`process_start` method of an earlier spider middleware,
overriding it. For example:
.. code-block:: python
async def process_start(self, start):
async for item_or_request in start:
yield item_or_request
You may yield the same type of objects as :meth:`~scrapy.Spider.start`.
To write spider middlewares that work on Scrapy versions lower than
2.13, define also a synchronous ``process_start_requests()`` method
that returns an iterable. For example:
.. code-block:: python
def process_start_requests(self, start, spider):
yield from start
System Message: ERROR/3 (<stdin>, line 97)
Unknown directive type "method".
.. method:: process_spider_input(response)
This method is called for each response that goes through the spider
middleware and into the spider, for processing.
:meth:`process_spider_input` should return ``None`` or raise an
exception.
If it returns ``None``, Scrapy will continue processing this response,
executing all other middlewares until, finally, the response is handed
to the spider for processing.
If it raises an exception, Scrapy won't bother calling any other spider
middleware :meth:`process_spider_input` and will call the request
errback if there is one, otherwise it will start the :meth:`process_spider_exception`
chain. The output of the errback is chained back in the other
direction for :meth:`process_spider_output` to process it, or
:meth:`process_spider_exception` if it raised an exception.
:param response: the response being processed
:type response: :class:`~scrapy.http.Response` object
System Message: ERROR/3 (<stdin>, line 119)
Unknown directive type "method".
.. method:: process_spider_output(response, result)
:async:
This method is an :term:`asynchronous generator` called with the
results from the spider after the spider has processed the response.
.. seealso:: :ref:`universal-spider-middleware`.
:param response: the response which generated this output from the
spider
:type response: :class:`~scrapy.http.Response` object
:param result: the results from the spider
:type result: an :term:`asynchronous iterable` of
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`
System Message: ERROR/3 (<stdin>, line 136)
Unknown directive type "method".
.. method:: process_spider_output_async(response, result)
:async:
Alternative name for :meth:`process_spider_output` used when
implementing a :ref:`universal spider middleware
<universal-spider-middleware>`.
System Message: ERROR/3 (<stdin>, line 143)
Unknown directive type "method".
.. method:: process_spider_exception(response, exception)
This method is called when a spider or :meth:`process_spider_output`
method (from a previous spider middleware) raises an exception.
:meth:`process_spider_exception` should return either ``None`` or an
iterable of :class:`~scrapy.Request` or :ref:`item <topics-items>`
objects.
If it returns ``None``, Scrapy will continue processing this exception,
executing any other :meth:`process_spider_exception` in the following
middleware components, until no middleware components are left and the
exception reaches the engine (where it's logged and discarded).
If it returns an iterable the :meth:`process_spider_output` pipeline
kicks in, starting from the next spider middleware, and no other
:meth:`process_spider_exception` will be called.
:param response: the response being processed when the exception was
raised
:type response: :class:`~scrapy.http.Response` object
:param exception: the exception raised
:type exception: :exc:`Exception` object
Universal spider middlewares
In Scrapy 2.6.3 and lower, process_spider_output() must be a synchronous generator.
To support those versions and higher Scrapy versions in the same middleware, rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()` method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define a synchronous process_spider_output() method to be used by 2.6.3 and lower versions.
System Message: ERROR/3 (<stdin>, line 177); backlink
Unknown interpreted text role "method".System Message: ERROR/3 (<stdin>, line 177); backlink
Unknown interpreted text role "method".For example:
System Message: WARNING/2 (<stdin>, line 185)
Cannot analyze code. Pygments package not found.
.. code-block:: python
class UniversalSpiderMiddleware:
async def process_spider_output_async(self, response, result):
async for r in result:
# ... do something with r
yield r
def process_spider_output(self, response, result):
for r in result:
# ... do something with r
yield r
Base class for custom spider middlewares
Scrapy provides a base class for custom spider middlewares. It's not required to use it but it can help with simplifying middleware implementations.
System Message: ERROR/3 (<stdin>, line 204)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.base
System Message: ERROR/3 (<stdin>, line 206)
Unknown directive type "autoclass".
.. autoclass:: BaseSpiderMiddleware :members:
Built-in spider middleware reference
This page describes all spider middleware components that come with Scrapy. For information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide <topics-spider-middleware>`.
System Message: ERROR/3 (<stdin>, line 214); backlink
Unknown interpreted text role "ref".For a list of the components enabled by default (and their orders) see the :setting:`SPIDER_MIDDLEWARES_BASE` setting.
System Message: ERROR/3 (<stdin>, line 218); backlink
Unknown interpreted text role "setting".DepthMiddleware
System Message: ERROR/3 (<stdin>, line 224)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.depth :synopsis: Depth Spider Middleware
DepthMiddleware is used for tracking the depth of each Request inside the site being scraped. It works by setting request.meta['depth'] = 0 whenever there is no value previously set (usually just the first Request) and incrementing it by 1 otherwise.
It can be used to limit the maximum depth to scrape, control Request priority based on their depth, and things like that.
The :class:`DepthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 237); backlink
Unknown interpreted text role "class".
:setting:`DEPTH_LIMIT` - The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed.
System Message: ERROR/3 (<stdin>, line 240); backlink
Unknown interpreted text role "setting".
:setting:`DEPTH_STATS_VERBOSE` - Whether to collect the number of requests for each depth.
System Message: ERROR/3 (<stdin>, line 242); backlink
Unknown interpreted text role "setting".
:setting:`DEPTH_PRIORITY` - Whether to prioritize the requests based on their depth.
System Message: ERROR/3 (<stdin>, line 244); backlink
Unknown interpreted text role "setting".
HttpErrorMiddleware
System Message: ERROR/3 (<stdin>, line 250)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.httperror :synopsis: HTTP Error Spider Middleware
Filter out unsuccessful (erroneous) HTTP responses so that spiders don't have to deal with them, which (most of the time) imposes an overhead, consumes more resources, and makes the spider logic more complex.
According to the HTTP standard, successful responses are those whose status codes are in the 200-300 range.
If you still want to process response codes outside that range, you can specify which response codes the spider is able to handle using the handle_httpstatus_list spider attribute or :setting:`HTTPERROR_ALLOWED_CODES` setting.
System Message: ERROR/3 (<stdin>, line 264); backlink
Unknown interpreted text role "setting".For example, if you want your spider to handle 404 responses you can do this:
System Message: WARNING/2 (<stdin>, line 272)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.spiders import CrawlSpider
class MySpider(CrawlSpider):
handle_httpstatus_list = [404]
System Message: ERROR/3 (<stdin>, line 280)
Unknown directive type "reqmeta".
.. reqmeta:: handle_httpstatus_list
System Message: ERROR/3 (<stdin>, line 282)
Unknown directive type "reqmeta".
.. reqmeta:: handle_httpstatus_all
The handle_httpstatus_list key of :attr:`Request.meta <scrapy.Request.meta>` can also be used to specify which response codes to allow on a per-request basis. You can also set the meta key handle_httpstatus_all to True if you want to allow any response code for a request, and False to disable the effects of the handle_httpstatus_all key.
System Message: ERROR/3 (<stdin>, line 284); backlink
Unknown interpreted text role "attr".Keep in mind, however, that it's usually a bad idea to handle non-200 responses, unless you really know what you're doing.
For more information see: HTTP Status Code Definitions.
HttpErrorMiddleware settings
System Message: ERROR/3 (<stdin>, line 300)
Unknown directive type "setting".
.. setting:: HTTPERROR_ALLOWED_CODES
HTTPERROR_ALLOWED_CODES
Default: []
Pass all responses with non-200 status codes contained in this list.
System Message: ERROR/3 (<stdin>, line 309)
Unknown directive type "setting".
.. setting:: HTTPERROR_ALLOW_ALL
HTTPERROR_ALLOW_ALL
Default: False
Pass all responses, regardless of its status code.
RefererMiddleware
System Message: ERROR/3 (<stdin>, line 322)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.referer :synopsis: Referer Spider Middleware
Populates Request Referer header, based on the URL of the Response which generated it.
RefererMiddleware settings
System Message: ERROR/3 (<stdin>, line 333)
Unknown directive type "setting".
.. setting:: REFERER_ENABLED
REFERER_ENABLED
Default: True
Whether to enable referer middleware.
System Message: ERROR/3 (<stdin>, line 342)
Unknown directive type "setting".
.. setting:: REFERRER_POLICY
REFERRER_POLICY
Default: 'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'
System Message: ERROR/3 (<stdin>, line 349)
Unknown directive type "reqmeta".
.. reqmeta:: referrer_policy
Referrer Policy to apply when populating Request "Referer" header.
Note
You can also set the Referrer Policy per request, using the special "referrer_policy" :ref:`Request.meta <topics-request-meta>` key, with the same acceptable values as for the REFERRER_POLICY setting.
System Message: ERROR/3 (<stdin>, line 354); backlink
Unknown interpreted text role "ref".Acceptable values for REFERRER_POLICY
either a path to a :class:`scrapy.spidermiddlewares.referer.ReferrerPolicy` subclass — a custom policy or one of the built-in ones (see classes below),
System Message: ERROR/3 (<stdin>, line 361); backlink
Unknown interpreted text role "class".
or one or more comma-separated standard W3C-defined string values,
or the special "scrapy-default".
| String value | Class name (as a string) |
|---|---|
| "scrapy-default" (default) | :class:`scrapy.spidermiddlewares.referer.DefaultReferrerPolicy` System Message: ERROR/3 (<stdin>, line 370); backlink Unknown interpreted text role "class". |
| "no-referrer" | :class:`scrapy.spidermiddlewares.referer.NoReferrerPolicy` System Message: ERROR/3 (<stdin>, line 371); backlink Unknown interpreted text role "class". |
| "no-referrer-when-downgrade" | :class:`scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy` System Message: ERROR/3 (<stdin>, line 372); backlink Unknown interpreted text role "class". |
| "same-origin" | :class:`scrapy.spidermiddlewares.referer.SameOriginPolicy` System Message: ERROR/3 (<stdin>, line 373); backlink Unknown interpreted text role "class". |
| "origin" | :class:`scrapy.spidermiddlewares.referer.OriginPolicy` System Message: ERROR/3 (<stdin>, line 374); backlink Unknown interpreted text role "class". |
| "strict-origin" | :class:`scrapy.spidermiddlewares.referer.StrictOriginPolicy` System Message: ERROR/3 (<stdin>, line 375); backlink Unknown interpreted text role "class". |
| "origin-when-cross-origin" | :class:`scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy` System Message: ERROR/3 (<stdin>, line 376); backlink Unknown interpreted text role "class". |
| "strict-origin-when-cross-origin" | :class:`scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy` System Message: ERROR/3 (<stdin>, line 377); backlink Unknown interpreted text role "class". |
| "unsafe-url" | :class:`scrapy.spidermiddlewares.referer.UnsafeUrlPolicy` System Message: ERROR/3 (<stdin>, line 378); backlink Unknown interpreted text role "class". |
System Message: ERROR/3 (<stdin>, line 380)
Unknown directive type "autoclass".
.. autoclass:: ReferrerPolicy
System Message: ERROR/3 (<stdin>, line 382)
Unknown directive type "autoclass".
.. autoclass:: DefaultReferrerPolicy
Warning
Scrapy's default referrer policy — just like "no-referrer-when-downgrade", the W3C-recommended value for browsers — will send a non-empty "Referer" header from any http(s):// to any https:// URL, even if the domain is different.
"same-origin" may be a better choice if you want to remove referrer information for cross-domain requests.
System Message: ERROR/3 (<stdin>, line 392)
Unknown directive type "autoclass".
.. autoclass:: NoReferrerPolicy
System Message: ERROR/3 (<stdin>, line 394)
Unknown directive type "autoclass".
.. autoclass:: NoReferrerWhenDowngradePolicy
Note
"no-referrer-when-downgrade" policy is the W3C-recommended default, and is used by major web browsers.
However, it is NOT Scrapy's default referrer policy (see :class:`DefaultReferrerPolicy`).
System Message: ERROR/3 (<stdin>, line 399); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 401)
Unknown directive type "autoclass".
.. autoclass:: SameOriginPolicy
System Message: ERROR/3 (<stdin>, line 403)
Unknown directive type "autoclass".
.. autoclass:: OriginPolicy
System Message: ERROR/3 (<stdin>, line 405)
Unknown directive type "autoclass".
.. autoclass:: StrictOriginPolicy
System Message: ERROR/3 (<stdin>, line 407)
Unknown directive type "autoclass".
.. autoclass:: OriginWhenCrossOriginPolicy
System Message: ERROR/3 (<stdin>, line 409)
Unknown directive type "autoclass".
.. autoclass:: StrictOriginWhenCrossOriginPolicy
System Message: ERROR/3 (<stdin>, line 411)
Unknown directive type "autoclass".
.. autoclass:: UnsafeUrlPolicy
Warning
"unsafe-url" policy is NOT recommended.
System Message: ERROR/3 (<stdin>, line 425)
Unknown directive type "setting".
.. setting:: REFERRER_POLICIES
REFERRER_POLICIES
System Message: ERROR/3 (<stdin>, line 430)
Unknown directive type "versionadded".
.. versionadded:: 2.14.2
Default: {}
A dictionary mapping policy names to import paths of :class:`scrapy.spidermiddlewares.referer.ReferrerPolicy` subclasses, or None to disable support for a given policy name.
System Message: ERROR/3 (<stdin>, line 434); backlink
Unknown interpreted text role "class".This allows overriding the policies triggered by the Referrer-Policy response header.
Use "" to override the policy for responses with no referrer policy.
StartSpiderMiddleware
System Message: ERROR/3 (<stdin>, line 448)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.start
System Message: ERROR/3 (<stdin>, line 450)
Unknown directive type "autoclass".
.. autoclass:: StartSpiderMiddleware
UrlLengthMiddleware
System Message: ERROR/3 (<stdin>, line 456)
Unknown directive type "module".
.. module:: scrapy.spidermiddlewares.urllength :synopsis: URL Length Spider Middleware
Filters out requests with URLs longer than URLLENGTH_LIMIT
The :class:`UrlLengthMiddleware` can be configured through the following settings (see the settings documentation for more info):
System Message: ERROR/3 (<stdin>, line 463); backlink
Unknown interpreted text role "class".
:setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
System Message: ERROR/3 (<stdin>, line 466); backlink
Unknown interpreted text role "setting".