15 KiB

Raw Blame History

Extensions

Extensions are :ref:`components <topics-components>` that allow inserting your own custom functionality into Scrapy.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Unlike other components, extensions do not have a specific role in Scrapy. They are “wildcard” components that can be used for anything that does not fit the role of any other type of component.

Loading and activating extensions

Extensions are loaded at startup by creating a single instance of the extension class per spider being run.

To enable an extension, add it to the :setting:`EXTENSIONS` setting. For example:

System Message: ERROR/3 (<stdin>, line 20); backlink

Unknown interpreted text role "setting".

System Message: WARNING/2 (<stdin>, line 23)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    EXTENSIONS = {
        "scrapy.extensions.corestats.CoreStats": 500,
        "scrapy.extensions.telnet.TelnetConsole": 500,
    }

:setting:`EXTENSIONS` is merged with :setting:`EXTENSIONS_BASE` (not meant to be overridden), and the priorities in the resulting value determine the loading order.

System Message: ERROR/3 (<stdin>, line 30); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 30); backlink

Unknown interpreted text role "setting".

As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting defines all extensions with the same order (0). However, you may need to carefully use priorities if you add an extension that depends on other extensions being already loaded.

System Message: ERROR/3 (<stdin>, line 34); backlink

Unknown interpreted text role "setting".

Writing your own extension

Each extension is a :ref:`component <topics-components>`.

System Message: ERROR/3 (<stdin>, line 43); backlink

Unknown interpreted text role "ref".

Typically, extensions connect to :ref:`signals <topics-signals>` and perform tasks triggered by them.

System Message: ERROR/3 (<stdin>, line 45); backlink

Unknown interpreted text role "ref".

Sample extension

Here we will implement a simple extension to illustrate the concepts described in the previous section. This extension will log a message every time:

a spider is opened
a spider is closed
a specific number of items are scraped

The extension will be enabled through the MYEXT_ENABLED setting and the number of items will be specified through the MYEXT_ITEMCOUNT setting.

Here is the code of such extension:

System Message: WARNING/2 (<stdin>, line 63)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import logging
    from scrapy import signals
    from scrapy.exceptions import NotConfigured

    logger = logging.getLogger(__name__)


    class SpiderOpenCloseLogging:
        def __init__(self, item_count):
            self.item_count = item_count
            self.items_scraped = 0

        @classmethod
        def from_crawler(cls, crawler):
            # first check if the extension should be enabled and raise
            # NotConfigured otherwise
            if not crawler.settings.getbool("MYEXT_ENABLED"):
                raise NotConfigured

            # get the number of items from settings
            item_count = crawler.settings.getint("MYEXT_ITEMCOUNT", 1000)

            # instantiate the extension object
            ext = cls(item_count)

            # connect the extension object to signals
            crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
            crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
            crawler.signals.connect(ext.item_scraped, signal=signals.item_scraped)

            # return the extension object
            return ext

        def spider_opened(self, spider):
            logger.info("opened spider %s", spider.name)

        def spider_closed(self, spider):
            logger.info("closed spider %s", spider.name)

        def item_scraped(self, item, spider):
            self.items_scraped += 1
            if self.items_scraped % self.item_count == 0:
                logger.info("scraped %d items", self.items_scraped)

Built-in extensions reference

General purpose extensions

Log Stats extension

System Message: ERROR/3 (<stdin>, line 121)

Unknown directive type "module".

.. module:: scrapy.extensions.logstats
   :synopsis: Basic stats logging

Log basic stats like crawled pages and scraped items.

Core Stats extension

System Message: ERROR/3 (<stdin>, line 131)

Unknown directive type "module".

.. module:: scrapy.extensions.corestats
   :synopsis: Core stats collection

Enable the collection of core statistics, provided the stats collection is enabled (see :ref:`topics-stats`).

System Message: ERROR/3 (<stdin>, line 136); backlink

Unknown interpreted text role "ref".

Log Count extension

System Message: ERROR/3 (<stdin>, line 144)

Unknown directive type "module".

.. module:: scrapy.extensions.logcount
   :synopsis: Basic stats logging

System Message: ERROR/3 (<stdin>, line 147)

Unknown directive type "autoclass".

.. autoclass:: LogCount

Telnet console extension

System Message: ERROR/3 (<stdin>, line 152)

Unknown directive type "module".

.. module:: scrapy.extensions.telnet
   :synopsis: Telnet console

Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging.

The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED` setting, and the server will listen in the port specified in :setting:`TELNETCONSOLE_PORT`.

System Message: ERROR/3 (<stdin>, line 160); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 160); backlink

Unknown interpreted text role "setting".

Memory usage extension

System Message: ERROR/3 (<stdin>, line 169)

Unknown directive type "module".

.. module:: scrapy.extensions.memusage
   :synopsis: Memory usage extension

Note

This extension does not work in Windows.

Monitors the memory used by the Scrapy process that runs the spider and:

sends a :signal:`memusage_warning_reached` signal when it exceeds :setting:`MEMUSAGE_WARNING_MB`

System Message: ERROR/3 (<stdin>, line 178); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 178); backlink

Unknown interpreted text role "setting".
closes the spider with the "memusage_exceeded" reason when it exceeds :setting:`MEMUSAGE_LIMIT_MB`

System Message: ERROR/3 (<stdin>, line 180); backlink

Unknown interpreted text role "setting".

This extension is enabled by the :setting:`MEMUSAGE_ENABLED` setting and can be configured with the following settings:

System Message: ERROR/3 (<stdin>, line 183); backlink

Unknown interpreted text role "setting".

:setting:`MEMUSAGE_LIMIT_MB`

System Message: ERROR/3 (<stdin>, line 186); backlink

Unknown interpreted text role "setting".
:setting:`MEMUSAGE_WARNING_MB`

System Message: ERROR/3 (<stdin>, line 187); backlink

Unknown interpreted text role "setting".
:setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS`

System Message: ERROR/3 (<stdin>, line 188); backlink

Unknown interpreted text role "setting".

Memory debugger extension

System Message: ERROR/3 (<stdin>, line 193)

Unknown directive type "module".

.. module:: scrapy.extensions.memdebug
   :synopsis: Memory debugger extension

An extension for debugging memory usage. It collects information about:

objects uncollected by the Python garbage collector
objects left alive that shouldn't. For more info, see :ref:`topics-leaks-trackrefs`

System Message: ERROR/3 (<stdin>, line 201); backlink

Unknown interpreted text role "ref".

To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The info will be stored in the stats.

System Message: ERROR/3 (<stdin>, line 203); backlink

Unknown interpreted text role "setting".

Spider state extension

System Message: ERROR/3 (<stdin>, line 211)

Unknown directive type "module".

.. module:: scrapy.extensions.spiderstate
   :synopsis: Spider state extension

Manages spider state data by loading it before a crawl and saving it after.

Give a value to the :setting:`JOBDIR` setting to enable this extension. When enabled, this extension manages the :attr:`~scrapy.Spider.state` attribute of your :class:`~scrapy.Spider` instance:

System Message: ERROR/3 (<stdin>, line 218); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 218); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 218); backlink

Unknown interpreted text role "class".

When your spider closes (:signal:`spider_closed`), the contents of its :attr:`~scrapy.Spider.state` attribute are serialized into a file named spider.state in the :setting:`JOBDIR` folder.

System Message: ERROR/3 (<stdin>, line 222); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 222); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 222); backlink

Unknown interpreted text role "setting".
When your spider opens (:signal:`spider_opened`), if a previously-generated spider.state file exists in the :setting:`JOBDIR` folder, it is loaded into the :attr:`~scrapy.Spider.state` attribute.

System Message: ERROR/3 (<stdin>, line 225); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 225); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 225); backlink

Unknown interpreted text role "attr".

For an example, see :ref:`topics-keeping-persistent-state-between-batches`.

System Message: ERROR/3 (<stdin>, line 230); backlink

Unknown interpreted text role "ref".

Close spider extension

System Message: ERROR/3 (<stdin>, line 235)

Unknown directive type "module".

.. module:: scrapy.extensions.closespider
   :synopsis: Close spider extension

Closes a spider automatically when some conditions are met, using a specific closing reason for each condition.

The conditions for closing a spider can be configured through the following settings:

:setting:`CLOSESPIDER_TIMEOUT`

System Message: ERROR/3 (<stdin>, line 246); backlink

Unknown interpreted text role "setting".
:setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`

System Message: ERROR/3 (<stdin>, line 247); backlink

Unknown interpreted text role "setting".
:setting:`CLOSESPIDER_ITEMCOUNT`

System Message: ERROR/3 (<stdin>, line 248); backlink

Unknown interpreted text role "setting".
:setting:`CLOSESPIDER_PAGECOUNT`

System Message: ERROR/3 (<stdin>, line 249); backlink

Unknown interpreted text role "setting".
:setting:`CLOSESPIDER_ERRORCOUNT`

System Message: ERROR/3 (<stdin>, line 250); backlink

Unknown interpreted text role "setting".

Note

When a certain closing condition is met, requests which are currently in the downloader queue (up to :setting:`CONCURRENT_REQUESTS` requests) are still processed.

System Message: ERROR/3 (<stdin>, line 254); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 258)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_TIMEOUT

CLOSESPIDER_TIMEOUT

Default: 0

An integer which specifies a number of seconds. If the spider remains open for more than that number of seconds, it will be automatically closed with the reason closespider_timeout. If zero (or non set), spiders won't be closed by timeout.

System Message: ERROR/3 (<stdin>, line 270)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_TIMEOUT_NO_ITEM

CLOSESPIDER_TIMEOUT_NO_ITEM

Default: 0

An integer which specifies a number of seconds. If the spider has not produced any items in the last number of seconds, it will be closed with the reason closespider_timeout_no_item. If zero (or non set), spiders won't be closed regardless if it hasn't produced any items.

System Message: ERROR/3 (<stdin>, line 282)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_ITEMCOUNT

CLOSESPIDER_ITEMCOUNT

Default: 0

An integer which specifies a number of items. If the spider scrapes more than that amount and those items are passed by the item pipeline, the spider will be closed with the reason closespider_itemcount. If zero (or non set), spiders won't be closed by number of passed items.

System Message: ERROR/3 (<stdin>, line 294)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_PAGECOUNT

CLOSESPIDER_PAGECOUNT

Default: 0

An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders won't be closed by number of crawled responses.

System Message: ERROR/3 (<stdin>, line 306)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_PAGECOUNT_NO_ITEM

CLOSESPIDER_PAGECOUNT_NO_ITEM

Default: 0

An integer which specifies the maximum number of consecutive responses to crawl without items scraped. If the spider crawls more consecutive responses than that and no items are scraped in the meantime, the spider will be closed with the reason closespider_pagecount_no_item. If zero (or not set), spiders won't be closed by number of crawled responses with no items.

System Message: ERROR/3 (<stdin>, line 319)

Unknown directive type "setting".

.. setting:: CLOSESPIDER_ERRORCOUNT

CLOSESPIDER_ERRORCOUNT

Default: 0

An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won't be closed by number of errors.

System Message: ERROR/3 (<stdin>, line 331)

Unknown directive type "module".

.. module:: scrapy.extensions.debug
   :synopsis: Extensions for debugging Scrapy

System Message: ERROR/3 (<stdin>, line 334)

Unknown directive type "module".

.. module:: scrapy.extensions.periodic_log
   :synopsis: Periodic stats logging

Periodic log extension

This extension periodically logs rich stat data as a JSON object:

2023-08-04 02:30:57 [scrapy.extensions.logstats] INFO: Crawled 976 pages (at 162 pages/min), scraped 925 items (at 161 items/min)
2023-08-04 02:30:57 [scrapy.extensions.periodic_log] INFO: {
    "delta": {
        "downloader/request_bytes": 55582,
        "downloader/request_count": 162,
        "downloader/request_method_count/GET": 162,
        "downloader/response_bytes": 618133,
        "downloader/response_count": 162,
        "downloader/response_status_count/200": 162,
        "item_scraped_count": 161
    },
    "stats": {
        "downloader/request_bytes": 338243,
        "downloader/request_count": 992,
        "downloader/request_method_count/GET": 992,
        "downloader/response_bytes": 3836736,
        "downloader/response_count": 976,
        "downloader/response_status_count/200": 976,
        "item_scraped_count": 925,
        "log_count/INFO": 21,
        "log_count/WARNING": 1,
        "scheduler/dequeued": 992,
        "scheduler/dequeued/memory": 992,
        "scheduler/enqueued": 1050,
        "scheduler/enqueued/memory": 1050
    },
    "time": {
        "elapsed": 360.008903,
        "log_interval": 60.0,
        "log_interval_real": 60.006694,
        "start_time": "2023-08-03 23:24:57",
        "utcnow": "2023-08-03 23:30:57"
    }
}

This extension logs the following configurable sections:

"delta" shows how some numeric stats have changed since the last stats log message.

The :setting:`PERIODIC_LOG_DELTA` setting determines the target stats. They must have int or float values.

System Message: ERROR/3 (<stdin>, line 384); backlink

Unknown interpreted text role "setting".
"stats" shows the current value of some stats.

The :setting:`PERIODIC_LOG_STATS` setting determines the target stats.

System Message: ERROR/3 (<stdin>, line 389); backlink

Unknown interpreted text role "setting".
"time" shows detailed timing data.

The :setting:`PERIODIC_LOG_TIMING_ENABLED` setting determines whether or not to show this section.

System Message: ERROR/3 (<stdin>, line 393); backlink

Unknown interpreted text role "setting".

This extension logs data at the start, then on a fixed time interval configurable through the :setting:`LOGSTATS_INTERVAL` setting, and finally right before the crawl ends.

System Message: ERROR/3 (<stdin>, line 396); backlink

Unknown interpreted text role "setting".

Example extension configuration:

System Message: WARNING/2 (<stdin>, line 403)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    custom_settings = {
        "LOG_LEVEL": "INFO",
        "PERIODIC_LOG_STATS": {
            "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
        },
        "PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
        "PERIODIC_LOG_TIMING_ENABLED": True,
        "EXTENSIONS": {
            "scrapy.extensions.periodic_log.PeriodicLog": 0,
        },
    }

System Message: ERROR/3 (<stdin>, line 417)

Unknown directive type "setting".

.. setting:: PERIODIC_LOG_DELTA

PERIODIC_LOG_DELTA

Default: None

"PERIODIC_LOG_DELTA": True - show deltas for all int and float stat values.
"PERIODIC_LOG_DELTA": {"include": ["downloader/", "scheduler/"]} - show deltas for stats with names containing any configured substring.
"PERIODIC_LOG_DELTA": {"exclude": ["downloader/"]} - show deltas for all stats with names not containing any configured substring.

System Message: ERROR/3 (<stdin>, line 428)

Unknown directive type "setting".

.. setting:: PERIODIC_LOG_STATS

PERIODIC_LOG_STATS

Default: None

"PERIODIC_LOG_STATS": True - show the current value of all stats.
"PERIODIC_LOG_STATS": {"include": ["downloader/", "scheduler/"]} - show current values for stats with names containing any configured substring.
"PERIODIC_LOG_STATS": {"exclude": ["downloader/"]} - show current values for all stats with names not containing any configured substring.

System Message: ERROR/3 (<stdin>, line 440)

Unknown directive type "setting".

.. setting:: PERIODIC_LOG_TIMING_ENABLED

PERIODIC_LOG_TIMING_ENABLED

Default: False

True enables logging of timing data (i.e. the "time" section).

Debugging extensions

Stack trace dump extension

Dumps information about the running process when a SIGQUIT or SIGUSR2 signal is received. The information dumped is the following:

engine status (using scrapy.utils.engine.get_engine_status())
live references (see :ref:`topics-leaks-trackrefs`)

System Message: ERROR/3 (<stdin>, line 462); backlink

Unknown interpreted text role "ref".
stack trace of all threads

After the stack trace and engine status is dumped, the Scrapy process continues running normally.

This extension only works on POSIX-compliant platforms (i.e. not Windows), because the SIGQUIT and SIGUSR2 signals are not available on Windows.

There are at least two ways to send Scrapy the SIGQUIT signal:

By pressing Ctrl-while a Scrapy process is running (Linux only?)
By running this command (assuming <pid> is the process id of the Scrapy process):
```
kill -QUIT <pid>
```

Debugger extension

Invokes a :doc:`Python debugger <library/pdb>` inside a running Scrapy process when a SIGUSR2 signal is received. After the debugger is exited, the Scrapy process continues running normally.

System Message: ERROR/3 (<stdin>, line 487); backlink

Unknown interpreted text role "doc".

This extension only works on POSIX-compliant platforms (i.e. not Windows).

</html>

15 KiB Raw Blame History

Extensions

Loading and activating extensions

Writing your own extension

Sample extension

Built-in extensions reference

General purpose extensions

Log Stats extension

Core Stats extension

Log Count extension

Telnet console extension

Memory usage extension

Memory debugger extension

Spider state extension

Close spider extension

CLOSESPIDER_TIMEOUT

CLOSESPIDER_TIMEOUT_NO_ITEM

CLOSESPIDER_ITEMCOUNT

CLOSESPIDER_PAGECOUNT

CLOSESPIDER_PAGECOUNT_NO_ITEM

CLOSESPIDER_ERRORCOUNT

Periodic log extension

PERIODIC_LOG_DELTA

PERIODIC_LOG_STATS

PERIODIC_LOG_TIMING_ENABLED

Debugging extensions

Stack trace dump extension

Debugger extension

15 KiB

Raw Blame History