RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/stats.rst

25 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Stats Collection

Scrapy provides a convenient facility for collecting stats in the form of key/value pairs, where values are often counters. The facility is called the Stats Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats` attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in the :ref:`topics-stats-usecases` section below.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

The Stats Collector API is always available, so you can always use it (to increment or set new stat keys), regardless of whether the stats collection is enabled or not. If it's disabled, the API will still work but it won't collect anything. This is aimed at simplifying the stats collector usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you're using the Stats Collector from.

Another feature of the Stats Collector is that it's very efficient (when enabled) and almost unnoticeable when disabled.

See :ref:`topics-stats-reference` below for the stats that Scrapy sets.

System Message: ERROR/3 (<stdin>, line 25); backlink

Unknown interpreted text role "ref".

Common Stats Collector uses

Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats` attribute. Here is an example of an extension that accesses stats:

System Message: ERROR/3 (<stdin>, line 32); backlink

Unknown interpreted text role "attr".

System Message: WARNING/2 (<stdin>, line 35)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class ExtensionThatAccessStats:
        def __init__(self, stats):
            self.stats = stats

        @classmethod
        def from_crawler(cls, crawler):
            return cls(crawler.stats)

Set stat value:

System Message: WARNING/2 (<stdin>, line 49)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    stats.set_value("hostname", socket.gethostname())

Increment stat value:

System Message: WARNING/2 (<stdin>, line 55)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    stats.inc_value("custom_count")

Set stat value only if greater than previous:

System Message: WARNING/2 (<stdin>, line 61)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    stats.max_value("max_items_scraped", value)

Set stat value only if lower than previous:

System Message: WARNING/2 (<stdin>, line 67)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    stats.min_value("min_free_memory_percent", value)

Get stat value:

System Message: WARNING/2 (<stdin>, line 73)

Cannot analyze code. Pygments package not found.

.. code-block:: pycon

    >>> stats.get_value("custom_count")
    1

Get all stats:

System Message: WARNING/2 (<stdin>, line 80)

Cannot analyze code. Pygments package not found.

.. code-block:: pycon

    >>> stats.get_stats()
    {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}

Available Stats Collectors

System Message: ERROR/3 (<stdin>, line 90)

Unknown directive type "currentmodule".

.. currentmodule:: scrapy.statscollectors

Besides the basic :class:`StatsCollector` there are other Stats Collectors available in Scrapy which extend the basic Stats Collector. You can select which Stats Collector to use through the :setting:`STATS_CLASS` setting. The default Stats Collector used is the :class:`MemoryStatsCollector`.

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "class".

MemoryStatsCollector

System Message: ERROR/3 (<stdin>, line 100)

Unknown directive type "autoclass".

.. autoclass:: MemoryStatsCollector
   :members:

DummyStatsCollector

System Message: ERROR/3 (<stdin>, line 106)

Unknown directive type "autoclass".

.. autoclass:: DummyStatsCollector

Built-in stats reference

Scrapy sets the following :ref:`stats <topics-stats>`. Components other than those built into Scrapy may set additional stats; see their documentation.

System Message: ERROR/3 (<stdin>, line 113); backlink

Unknown interpreted text role "ref".

Stat keys that contain a {placeholder} below stand for a family of stats, one per actual value of the placeholder.

Note

Most stats are set by a specific :ref:`component <topics-components>`, and are only present if that component is enabled and its code path is reached. A stat that is missing from :meth:`~scrapy.statscollectors.StatsCollector.get_stats` output is equivalent to a counter of 0.

System Message: ERROR/3 (<stdin>, line 119); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 119); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 125)

Unknown directive type "stat".

.. stat:: depth/request_ignored_count
depth/request_ignored_count

Number of requests dropped for exceeding :setting:`DEPTH_LIMIT`.

System Message: ERROR/3 (<stdin>, line 128); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`.

System Message: ERROR/3 (<stdin>, line 130); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 132)

Unknown directive type "stat".

.. stat:: downloader/exception_count
downloader/exception_count

Number of exceptions raised while downloading requests.

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 137); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 139)

Unknown directive type "stat".

.. stat:: downloader/exception_type_count/{exception_type}
downloader/exception_type_count/{exception_type}

Number of exceptions raised while downloading requests, per exception type, where {exception_type} is the import path of the exception class, e.g. twisted.internet.error.DNSLookupError.

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 146); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 148)

Unknown directive type "stat".

.. stat:: downloader/request_bytes
downloader/request_bytes

Total size, in bytes, of the requests sent, counting the request line, the headers and the body. As with :stat:`downloader/request_count`, requests served from the cache are also counted.

System Message: ERROR/3 (<stdin>, line 151); backlink

Unknown interpreted text role "stat".

It is an approximation, reconstructed from each :class:`~scrapy.Request` object instead of measured on the wire, so it does not account for the actual bytes that the :ref:`download handler <topics-download-handlers>` sends, e.g. transport-level overhead.

System Message: ERROR/3 (<stdin>, line 155); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 155); backlink

Unknown interpreted text role "ref".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 160); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 162)

Unknown directive type "stat".

.. stat:: downloader/request_count
downloader/request_count

Number of requests sent.

Requests that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache are also counted, even though they are never sent, because it handles requests after :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 167); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 167); backlink

Unknown interpreted text role "class".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 174)

Unknown directive type "stat".

.. stat:: downloader/request_method_count/{method}
downloader/request_method_count/{method}

Number of requests sent, per HTTP method, e.g. GET or POST. As with :stat:`downloader/request_count`, requests served from the cache are also counted.

System Message: ERROR/3 (<stdin>, line 177); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 181); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 183)

Unknown directive type "stat".

.. stat:: downloader/response_bytes
downloader/response_bytes

Total size, in bytes, of the responses received, counting the status line, the headers and the body. It covers the same responses as :stat:`downloader/response_count`.

System Message: ERROR/3 (<stdin>, line 186); backlink

Unknown interpreted text role "stat".

The body is counted as received, i.e. still compressed for responses that used Content-Encoding, because :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` handles responses before :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware` decompresses them. See :stat:`httpcompression/response_bytes` for decompressed sizes.

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 198); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 200)

Unknown directive type "stat".

.. stat:: downloader/response_count
downloader/response_count

Number of responses received.

It counts responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache, even though they do not come from the network, and responses that a downloader middleware consumes before they reach your spider, e.g. redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` turns into new requests. Compare with :stat:`response_received_count`.

System Message: ERROR/3 (<stdin>, line 205); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 205); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 205); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 211); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 213)

Unknown directive type "stat".

.. stat:: downloader/response_status_count/{status_code}
downloader/response_status_count/{status_code}

Number of responses received, per HTTP status code, e.g. 200 or 404. It covers the same responses as :stat:`downloader/response_count`.

System Message: ERROR/3 (<stdin>, line 216); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.

System Message: ERROR/3 (<stdin>, line 219); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 221)

Unknown directive type "stat".

.. stat:: dupefilter/filtered
dupefilter/filtered

Number of requests dropped as duplicates.

Set by :class:`~scrapy.dupefilters.RFPDupeFilter`.

System Message: ERROR/3 (<stdin>, line 226); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 228)

Unknown directive type "stat".

.. stat:: elapsed_time_seconds
elapsed_time_seconds

Time, as a :class:`float`, in seconds, between the :signal:`spider_opened` and the :signal:`spider_closed` signals.

System Message: ERROR/3 (<stdin>, line 231); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 231); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 231); backlink

Unknown interpreted text role "signal".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 234); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 236)

Unknown directive type "stat".

.. stat:: feedexport/failed_count/{storage}
feedexport/failed_count/{storage}

Number of :ref:`feeds <topics-feed-exports>` that could not be stored, per :ref:`storage backend <topics-feed-storage-backends>`, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.

System Message: ERROR/3 (<stdin>, line 239); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 239); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 243)

Unknown directive type "stat".

.. stat:: feedexport/success_count/{storage}
feedexport/success_count/{storage}

Number of :ref:`feeds <topics-feed-exports>` stored successfully, per :ref:`storage backend <topics-feed-storage-backends>`, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.

System Message: ERROR/3 (<stdin>, line 246); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 246); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 250)

Unknown directive type "stat".

.. stat:: file_count
file_count

Number of files handled by the :ref:`media pipelines <topics-media-pipeline>`.

System Message: ERROR/3 (<stdin>, line 253); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 256)

Unknown directive type "stat".

.. stat:: file_status_count/{status}
file_status_count/{status}

Number of files handled by the :ref:`media pipelines <topics-media-pipeline>`, per status, where {status} is one of:

System Message: ERROR/3 (<stdin>, line 259); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 271)

Unknown directive type "stat".

.. stat:: finish_reason
finish_reason

String indicating why the crawl finished. It matches the reason argument of the :signal:`spider_closed` signal.

System Message: ERROR/3 (<stdin>, line 274); backlink

Unknown interpreted text role "signal".

Scrapy uses the following reasons:

Third-party components and your own code may use any other reason, e.g. by raising :exc:`~scrapy.exceptions.CloseSpider` with it.

System Message: ERROR/3 (<stdin>, line 309); backlink

Unknown interpreted text role "exc".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 312); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 314)

Unknown directive type "stat".

.. stat:: finish_time
finish_time

Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when the :signal:`spider_closed` signal was sent.

System Message: ERROR/3 (<stdin>, line 317); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 317); backlink

Unknown interpreted text role "signal".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 320); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 322)

Unknown directive type "stat".

.. stat:: httpcache/errorrecovery
httpcache/errorrecovery

Number of times that a stale cached response was used because downloading a fresh response raised an exception.

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 328); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 330)

Unknown directive type "stat".

.. stat:: httpcache/firsthand
httpcache/firsthand

Number of responses that were downloaded without a matching cache entry to validate against, i.e. responses for requests counted in :stat:`httpcache/miss`.

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "stat".

It is lower than :stat:`httpcache/miss` when some of those requests yield no response, either because they are dropped (see :stat:`httpcache/ignore`) or because their download fails.

System Message: ERROR/3 (<stdin>, line 337); backlink

Unknown interpreted text role "stat".

System Message: ERROR/3 (<stdin>, line 337); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 343)

Unknown directive type "stat".

.. stat:: httpcache/hit
httpcache/hit

Number of requests served from the cache.

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 348); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 350)

Unknown directive type "stat".

.. stat:: httpcache/ignore
httpcache/ignore

Number of requests dropped because they were not in the cache and :setting:`HTTPCACHE_IGNORE_MISSING` is True.

System Message: ERROR/3 (<stdin>, line 353); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 356); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 358)

Unknown directive type "stat".

.. stat:: httpcache/invalidate
httpcache/invalidate

Number of times that a cached response failed validation and was replaced with a freshly downloaded response.

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 364); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 366)

Unknown directive type "stat".

.. stat:: httpcache/miss
httpcache/miss

Number of requests for which no cache entry could be read, either because there was none or because reading it failed, in which case the request is also counted in :stat:`httpcache/retrieve_error`. Those requests are downloaded (see :stat:`httpcache/firsthand`), or dropped if :setting:`HTTPCACHE_IGNORE_MISSING` is True (see :stat:`httpcache/ignore`).

System Message: ERROR/3 (<stdin>, line 369); backlink

Unknown interpreted text role "stat".

System Message: ERROR/3 (<stdin>, line 369); backlink

Unknown interpreted text role "stat".

System Message: ERROR/3 (<stdin>, line 369); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 369); backlink

Unknown interpreted text role "stat".

Requests with a stale cache entry are not counted here; see :stat:`httpcache/revalidate` and :stat:`httpcache/invalidate`.

System Message: ERROR/3 (<stdin>, line 376); backlink

Unknown interpreted text role "stat".

System Message: ERROR/3 (<stdin>, line 376); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 379); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 381)

Unknown directive type "stat".

.. stat:: httpcache/retrieve_error
httpcache/retrieve_error

Number of cache entries that could not be read, and hence were treated as cache misses. Those requests are also counted in :stat:`httpcache/miss`.

System Message: ERROR/3 (<stdin>, line 384); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 387); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 389)

Unknown directive type "stat".

.. stat:: httpcache/revalidate
httpcache/revalidate

Number of times that a cached response was successfully validated against the target server, and hence used instead of the fresh response.

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 395); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 397)

Unknown directive type "stat".

.. stat:: httpcache/store
httpcache/store

Number of responses stored in the cache.

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 402); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 404)

Unknown directive type "stat".

.. stat:: httpcache/uncacheable
httpcache/uncacheable

Number of responses not stored in the cache because the :setting:`HTTPCACHE_POLICY` did not allow it.

System Message: ERROR/3 (<stdin>, line 407); backlink

Unknown interpreted text role "setting".

Every response considered for caching is counted either here or in :stat:`httpcache/store`, so httpcache/store + httpcache/uncacheable equals httpcache/firsthand + httpcache/invalidate.

System Message: ERROR/3 (<stdin>, line 410); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.

System Message: ERROR/3 (<stdin>, line 414); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 416)

Unknown directive type "stat".

.. stat:: httpcompression/response_bytes
httpcompression/response_bytes

Total size, in bytes, of decompressed response bodies, counting only the body and only responses that were actually decompressed. Compare with :stat:`downloader/response_bytes`.

System Message: ERROR/3 (<stdin>, line 419); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.

System Message: ERROR/3 (<stdin>, line 423); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 426)

Unknown directive type "stat".

.. stat:: httpcompression/response_count
httpcompression/response_count

Number of decompressed responses.

Set by :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.

System Message: ERROR/3 (<stdin>, line 431); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 434)

Unknown directive type "stat".

.. stat:: httperror/response_ignored_count
httperror/response_ignored_count

Number of responses dropped because of their HTTP status code.

Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.

System Message: ERROR/3 (<stdin>, line 439); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 441)

Unknown directive type "stat".

.. stat:: httperror/response_ignored_status_count/{status_code}
httperror/response_ignored_status_count/{status_code}

Number of responses dropped because of their HTTP status code, per HTTP status code, e.g. 404.

Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.

System Message: ERROR/3 (<stdin>, line 447); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 449)

Unknown directive type "stat".

.. stat:: item_dropped_count
item_dropped_count

Number of items dropped by an :ref:`item pipeline <topics-item-pipeline>`, i.e. number of times that the :signal:`item_dropped` signal was sent.

System Message: ERROR/3 (<stdin>, line 452); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 452); backlink

Unknown interpreted text role "signal".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 456); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 458)

Unknown directive type "stat".

.. stat:: item_dropped_reasons_count/{exception}
item_dropped_reasons_count/{exception}

Number of items dropped, per exception, where {exception} is the class name of the exception that caused the item to be dropped.

Only :exc:`~scrapy.exceptions.DropItem` and its subclasses drop items, and each one is counted under its own class name, e.g. item_dropped_reasons_count/DropItem for :exc:`~scrapy.exceptions.DropItem` itself and item_dropped_reasons_count/MyDropItem for a MyDropItem subclass of it. Any other exception raised by an :ref:`item pipeline <topics-item-pipeline>` triggers the :signal:`item_error` signal instead of :signal:`item_dropped`, and is not counted here or in :stat:`item_dropped_count`.

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "signal".

System Message: ERROR/3 (<stdin>, line 464); backlink

Unknown interpreted text role "stat".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 474); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 476)

Unknown directive type "stat".

.. stat:: item_scraped_count
item_scraped_count

Number of items that passed all :ref:`item pipelines <topics-item-pipeline>`, i.e. number of times that the :signal:`item_scraped` signal was sent.

System Message: ERROR/3 (<stdin>, line 479); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 479); backlink

Unknown interpreted text role "signal".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 483); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 485)

Unknown directive type "stat".

.. stat:: items_per_minute
items_per_minute

Average number of items scraped per minute during the crawl.

It is None if the crawl took less than a minute.

Set by :class:`~scrapy.extensions.logstats.LogStats`.

System Message: ERROR/3 (<stdin>, line 492); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 494)

Unknown directive type "stat".

.. stat:: log_count/{level}
log_count/{level}

Number of log messages, per logging level name, e.g. INFO or WARNING.

Only messages that the :setting:`LOG_LEVEL` setting allows are counted.

System Message: ERROR/3 (<stdin>, line 500); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.extensions.logcount.LogCount`.

System Message: ERROR/3 (<stdin>, line 502); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 504)

Unknown directive type "stat".

.. stat:: memdebug/gc_garbage_count
memdebug/gc_garbage_count

Number of objects in :data:`gc.garbage` when the spider is closed.

System Message: ERROR/3 (<stdin>, line 507); backlink

Unknown interpreted text role "data".

Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires :setting:`MEMDEBUG_ENABLED` to be True.

System Message: ERROR/3 (<stdin>, line 509); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 509); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 512)

Unknown directive type "stat".

.. stat:: memdebug/live_refs/{cls}
memdebug/live_refs/{cls}

Number of live objects of class {cls} when the spider is closed, as reported by :ref:`trackref <topics-leaks-trackrefs>`, e.g. memdebug/live_refs/HtmlResponse.

System Message: ERROR/3 (<stdin>, line 515); backlink

Unknown interpreted text role "ref".

Only set for classes with at least 1 live object.

Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires :setting:`MEMDEBUG_ENABLED` to be True.

System Message: ERROR/3 (<stdin>, line 521); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 521); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 524)

Unknown directive type "stat".

.. stat:: memusage/limit_reached
memusage/limit_reached

1 if memory usage exceeded :setting:`MEMUSAGE_LIMIT_MB`, which also stops the crawl.

System Message: ERROR/3 (<stdin>, line 527); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.

System Message: ERROR/3 (<stdin>, line 530); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 532)

Unknown directive type "stat".

.. stat:: memusage/max
memusage/max

Maximum peak memory usage, in bytes, observed during the crawl.

Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.

System Message: ERROR/3 (<stdin>, line 537); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 539)

Unknown directive type "stat".

.. stat:: memusage/startup
memusage/startup

Peak memory usage, in bytes, when the engine started.

Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.

System Message: ERROR/3 (<stdin>, line 544); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 546)

Unknown directive type "stat".

.. stat:: memusage/warning_reached
memusage/warning_reached

1 if memory usage exceeded :setting:`MEMUSAGE_WARNING_MB`.

System Message: ERROR/3 (<stdin>, line 549); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.

System Message: ERROR/3 (<stdin>, line 551); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 553)

Unknown directive type "stat".

.. stat:: offsite/domains
offsite/domains

Number of distinct domains for which at least 1 request was dropped for being offsite.

Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.

System Message: ERROR/3 (<stdin>, line 559); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 561)

Unknown directive type "stat".

.. stat:: offsite/filtered
offsite/filtered

Number of requests dropped for being offsite.

Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.

System Message: ERROR/3 (<stdin>, line 566); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 568)

Unknown directive type "stat".

.. stat:: request_depth_count/{depth}
request_depth_count/{depth}

Number of requests scheduled at depth {depth}, e.g. request_depth_count/2.

Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`, which requires :setting:`DEPTH_STATS_VERBOSE` to be True for this stat.

System Message: ERROR/3 (<stdin>, line 574); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 574); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 577)

Unknown directive type "stat".

.. stat:: request_depth_max
request_depth_max

Maximum depth reached.

Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`.

System Message: ERROR/3 (<stdin>, line 582); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 584)

Unknown directive type "stat".

.. stat:: response_received_count
response_received_count

Number of responses received, i.e. number of times that the :signal:`response_received` signal was sent.

System Message: ERROR/3 (<stdin>, line 587); backlink

Unknown interpreted text role "signal".

Unlike :stat:`downloader/response_count`, it does not count responses that a downloader middleware consumes before they reach the engine, e.g. redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` turns into new requests. Both count responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache.

System Message: ERROR/3 (<stdin>, line 590); backlink

Unknown interpreted text role "stat".

System Message: ERROR/3 (<stdin>, line 590); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 590); backlink

Unknown interpreted text role "class".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 596); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 598)

Unknown directive type "stat".

.. stat:: responses_per_minute
responses_per_minute

Average number of responses received per minute during the crawl.

It is None if the crawl took less than a minute.

Set by :class:`~scrapy.extensions.logstats.LogStats`.

System Message: ERROR/3 (<stdin>, line 605); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 607)

Unknown directive type "stat".

.. stat:: retry/count
retry/count

Number of requests retried.

Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.

System Message: ERROR/3 (<stdin>, line 612); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 612); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 615)

Unknown directive type "stat".

.. stat:: retry/max_reached
retry/max_reached

Number of requests that were not retried because they had already been retried :setting:`RETRY_TIMES` times.

System Message: ERROR/3 (<stdin>, line 618); backlink

Unknown interpreted text role "setting".

Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.

System Message: ERROR/3 (<stdin>, line 621); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 621); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 624)

Unknown directive type "stat".

.. stat:: retry/reason_count/{reason}
retry/reason_count/{reason}

Number of requests retried, per reason, e.g. retry/reason_count/twisted.internet.error.TimeoutError or retry/reason_count/504 Gateway Time-out.

Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.

System Message: ERROR/3 (<stdin>, line 631); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 631); backlink

Unknown interpreted text role "class".

Note

Code calling :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` may pass a custom stats_base_key, in which case retry is replaced with that key in the 3 stats above.

System Message: ERROR/3 (<stdin>, line 634); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 639)

Unknown directive type "stat".

.. stat:: robotstxt/exception_count/{exception_type}
robotstxt/exception_count/{exception_type}

Number of exceptions raised while downloading robots.txt files, per exception type, where {exception_type} is the string representation of the exception class, e.g. <class 'twisted.internet.error.DNSLookupError'>.

Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.

System Message: ERROR/3 (<stdin>, line 647); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 650)

Unknown directive type "stat".

.. stat:: robotstxt/forbidden
robotstxt/forbidden

Number of requests dropped for being disallowed by robots.txt.

Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.

System Message: ERROR/3 (<stdin>, line 655); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 658)

Unknown directive type "stat".

.. stat:: robotstxt/request_count
robotstxt/request_count

Number of robots.txt files requested, i.e. 1 per network location for which at least 1 request was sent.

Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.

System Message: ERROR/3 (<stdin>, line 664); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 667)

Unknown directive type "stat".

.. stat:: robotstxt/response_count
robotstxt/response_count

Number of robots.txt responses received.

Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.

System Message: ERROR/3 (<stdin>, line 672); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 675)

Unknown directive type "stat".

.. stat:: robotstxt/response_status_count/{status_code}
robotstxt/response_status_count/{status_code}

Number of robots.txt responses received, per HTTP status code, e.g. 404.

Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.

System Message: ERROR/3 (<stdin>, line 681); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 684)

Unknown directive type "stat".

.. stat:: scheduler/dequeued
scheduler/dequeued

Number of requests read from the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 687); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 689)

Unknown directive type "stat".

.. stat:: scheduler/dequeued/disk
scheduler/dequeued/disk

Number of requests read from the disk queue of the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 692); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 695)

Unknown directive type "stat".

.. stat:: scheduler/dequeued/memory
scheduler/dequeued/memory

Number of requests read from the memory queue of the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 698); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 701)

Unknown directive type "stat".

.. stat:: scheduler/enqueued
scheduler/enqueued

Number of requests stored into the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 704); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 706)

Unknown directive type "stat".

.. stat:: scheduler/enqueued/disk
scheduler/enqueued/disk

Number of requests stored into the disk queue of the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 709); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 712)

Unknown directive type "stat".

.. stat:: scheduler/enqueued/memory
scheduler/enqueued/memory

Number of requests stored into the memory queue of the :ref:`scheduler <topics-scheduler>`.

System Message: ERROR/3 (<stdin>, line 715); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 718)

Unknown directive type "stat".

.. stat:: scheduler/unserializable
scheduler/unserializable

Number of requests that could not be stored into the disk queue of the :ref:`scheduler <topics-scheduler>` because they could not be :ref:`serialized <request-serialization>`, and hence were stored into the memory queue instead.

System Message: ERROR/3 (<stdin>, line 721); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 721); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 726)

Unknown directive type "stat".

.. stat:: spider_exceptions/count
spider_exceptions/count

Number of unhandled exceptions raised by spider callbacks or by :meth:`~scrapy.Spider.start`.

System Message: ERROR/3 (<stdin>, line 729); backlink

Unknown interpreted text role "meth".

Set by the :ref:`engine <topics-architecture>` and the :ref:`scraper <topics-architecture>`.

System Message: ERROR/3 (<stdin>, line 732); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 732); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 735)

Unknown directive type "stat".

.. stat:: spider_exceptions/{exception}
spider_exceptions/{exception}

Same as :stat:`spider_exceptions/count`, per exception, where {exception} is the class name of the exception, e.g. spider_exceptions/ValueError.

System Message: ERROR/3 (<stdin>, line 738); backlink

Unknown interpreted text role "stat".

Set by the :ref:`engine <topics-architecture>` and the :ref:`scraper <topics-architecture>`.

System Message: ERROR/3 (<stdin>, line 742); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 742); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 745)

Unknown directive type "stat".

.. stat:: start_time
start_time

Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when the :signal:`spider_opened` signal was sent.

System Message: ERROR/3 (<stdin>, line 748); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 748); backlink

Unknown interpreted text role "signal".

Set by :class:`~scrapy.extensions.corestats.CoreStats`.

System Message: ERROR/3 (<stdin>, line 751); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 753)

Unknown directive type "stat".

.. stat:: urllength/request_ignored_count
urllength/request_ignored_count

Number of requests dropped for having a URL longer than :setting:`URLLENGTH_LIMIT`.

System Message: ERROR/3 (<stdin>, line 756); backlink

Unknown interpreted text role "setting".

Set by :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`.

System Message: ERROR/3 (<stdin>, line 759); backlink

Unknown interpreted text role "class".
</html>