11 KiB

Raw Blame History

Stats Collection

Overview

Scrapy provides a convenient service for collecting stats in the form of key/values, both globally and per spider. It's called the Stats Collector, and it's a singleton which can be imported and used quickly, as illustrated by the examples in the :ref:`topics-stats-usecases` section below.

System Message: ERROR/3 (<stdin>, line 10); backlink

Unknown interpreted text role "ref".

The stats collection is enabled by default but can be disabled through the :setting:`STATS_ENABLED` setting.

System Message: ERROR/3 (<stdin>, line 15); backlink

Unknown interpreted text role "setting".

However, the Stats Collector is always available, so you can always import it in your module and use its API (to increment or set new stat keys), regardless of whether the stats collection is enabled or not. If it's disabled, the API will still work but it won't collect anything. This is aimed at simplifying the stats collector usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you're using the Stats Collector from.

Another feature of the Stats Collector is that it's very efficient (when enabled) and extremely efficient (almost unnoticeable) when disabled.

The Stats Collector keeps one stats table per open spider and one global stats table. You can't set or get stats from a closed spider, but the spider-specific stats table is automatically opened when the spider is opened, and closed when the spider is closed.

Common Stats Collector uses

Import the stats collector:

from scrapy.stats import stats

Set global stat value:

stats.set_value('hostname', socket.gethostname())

Increment global stat value:

stats.inc_value('spiders_crawled')

Set global stat value only if greater than previous:

stats.max_value('max_items_scraped', value)

Set global stat value only if lower than previous:

stats.min_value('min_free_memory_percent', value)

Get global stat value:

>>> stats.get_value('spiders_crawled')
8

Get all global stats (ie. not particular to any spider):

>>> stats.get_stats()
{'hostname': 'localhost', 'spiders_crawled': 8}

Set spider specific stat value (spider stats must be opened first, but this task is handled automatically by the Scrapy engine):

stats.set_value('start_time', datetime.now(), spider=some_spider)

Where some_spider is a :class:`~scrapy.spider.BaseSpider` object.

System Message: ERROR/3 (<stdin>, line 74); backlink

Unknown interpreted text role "class".

Increment spider-specific stat value:

stats.inc_value('pages_crawled', spider=some_spider)

Set spider-specific stat value only if greater than previous:

stats.max_value('max_items_scraped', value, spider=some_spider)

Set spider-specific stat value only if lower than previous:

stats.min_value('min_free_memory_percent', value, spider=some_spider)

Get spider-specific stat value:

>>> stats.get_value('pages_crawled', spider=some_spider)
1238

Get all stats from a given spider:

>>> stats.get_stats('pages_crawled', spider=some_spider)
{'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}

Stats Collector API

There are several Stats Collectors available under the :mod:`scrapy.stats.collector` module and they all implement the Stats Collector API defined by the :class:`~scrapy.stats.collector.StatsCollector` class (which they all inherit from).

System Message: ERROR/3 (<stdin>, line 103); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 103); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 108)

Unknown directive type "module".

.. module:: scrapy.stats.collector
   :synopsis: Basic Stats Collectors

System Message: ERROR/3 (<stdin>, line 113)