From 7701e590fbc5ac4d5da8512b07dc4e81e0d9c6c1 Mon Sep 17 00:00:00 2001 From: Rohit Kumar Singh <145501871+Rohitkr117@users.noreply.github.com> Date: Sat, 2 Nov 2024 11:15:27 +0530 Subject: [PATCH] Documentation added for Spider State in extensions.rst (#6522) * Documentation added for Spider State in extensions.rst * Made correction in documentation for Spiderstate * Added appropriate intro for Spider state extension * Added reference for spiderstate extension * Added Spiderstate extension hyperlink refrence in jobs.rst --- docs/topics/extensions.rst | 26 ++++++++++++++++++++++++++ docs/topics/jobs.rst | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 7b34a19d5..9cbc9663d 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -243,6 +243,32 @@ An extension for debugging memory usage. It collects information about: To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The info will be stored in the stats. +.. _topics-extensions-ref-spiderstate: + +Spider state extension +~~~~~~~~~~~~~~~~~~~~~~ + +.. module:: scrapy.extensions.spiderstate + :synopsis: Spider state extension + +.. class:: SpiderState + +Manages spider state data by loading it before a crawl and saving it after. + +Give a value to the :setting:`JOBDIR` setting to enable this extension. +When enabled, this extension manages the :attr:`~scrapy.Spider.state` +attribute of your :class:`~scrapy.Spider` instance: + +- When your spider closes (:signal:`spider_closed`), the contents of its + :attr:`~scrapy.Spider.state` attribute are serialized into a file named + ``spider.state`` in the :setting:`JOBDIR` folder. +- When your spider opens (:signal:`spider_opened`), if a previously-generated + ``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loaded + into the :attr:`~scrapy.Spider.state` attribute. + + +For an example, see :ref:`topics-keeping-persistent-state-between-batches`. + Close spider extension ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index c7fc1ea48..0e705dc64 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -46,7 +46,7 @@ Keeping persistent state between batches Sometimes you'll want to keep some persistent spider state between pause/resume batches. You can use the ``spider.state`` attribute for that, which should be a -dict. There's a built-in extension that takes care of serializing, storing and +dict. There's :ref:`a built-in extension ` that takes care of serializing, storing and loading that attribute from the job directory, when the spider starts and stops.