From 8288f78a39939dbffea467bf110c64e005847117 Mon Sep 17 00:00:00 2001 From: djunzu Date: Mon, 16 Oct 2017 21:34:37 -0200 Subject: [PATCH] Add note about request.meta['depth'] in DepthMiddleware --- docs/topics/spider-middleware.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 9a0ccd0c1..a2d2556c5 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -188,9 +188,13 @@ DepthMiddleware .. class:: DepthMiddleware - DepthMiddleware is a scrape middleware used for tracking the depth of each - Request inside the site being scraped. It can be used to limit the maximum - depth to scrape or things like that. + DepthMiddleware is used for tracking the depth of each Request inside the + site being scraped. It works by setting `request.meta['depth'] = 0` whenever + there is no value previously set (usually just the first Request) and + incrementing it by 1 otherwise. + + It can be used to limit the maximum depth to scrape, control Request + priority based on their depth, and things like that. The :class:`DepthMiddleware` can be configured through the following settings (see the settings documentation for more info):