RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Documentation added for Spider State in extensions.rst (#6522) 

* Documentation added for Spider State in extensions.rst

* Made correction in documentation for Spiderstate

* Added appropriate intro for Spider state extension

* Added reference for spiderstate extension

* Added Spiderstate extension hyperlink refrence in jobs.rst
This commit is contained in:
Rohit Kumar Singh 2024-11-02 11:15:27 +05:30 committed by GitHub
parent d85c39f5bc
commit 7701e590fb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 27 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
docs/topics/extensions.rst
View File

@ -243,6 +243,32 @@ An extension for debugging memory usage. It collects information about:
To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The
info will be stored in the stats.
.. _topics-extensions-ref-spiderstate:
Spider state extension
~~~~~~~~~~~~~~~~~~~~~~
.. module:: scrapy.extensions.spiderstate
:synopsis: Spider state extension
.. class:: SpiderState
Manages spider state data by loading it before a crawl and saving it after.
Give a value to the :setting:`JOBDIR` setting to enable this extension.
When enabled, this extension manages the :attr:`~scrapy.Spider.state`
attribute of your :class:`~scrapy.Spider` instance:
- When your spider closes (:signal:`spider_closed`), the contents of its
:attr:`~scrapy.Spider.state` attribute are serialized into a file named
``spider.state`` in the :setting:`JOBDIR` folder.
- When your spider opens (:signal:`spider_opened`), if a previously-generated
``spider.state`` file exists in the :setting:`JOBDIR` folder, it is loaded
into the :attr:`~scrapy.Spider.state` attribute.
For an example, see :ref:`topics-keeping-persistent-state-between-batches`.
Close spider extension
~~~~~~~~~~~~~~~~~~~~~~

2
docs/topics/jobs.rst
View File

@ -46,7 +46,7 @@ Keeping persistent state between batches
Sometimes you'll want to keep some persistent spider state between pause/resume
batches. You can use the ``spider.state`` attribute for that, which should be a
dict. There's a built-in extension that takes care of serializing, storing and
dict. There's :ref:`a built-in extension <topics-extensions-ref-spiderstate>` that takes care of serializing, storing and
loading that attribute from the job directory, when the spider starts and
stops.