24 KiB
Stats Collection
Scrapy provides a convenient facility for collecting stats in the form of key/values, where values are often counters. The facility is called the Stats Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats` attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in the :ref:`topics-stats-usecases` section below.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".The Stats Collector API is always available, so you can always use it (to increment or set new stat keys), regardless of whether the stats collection is enabled or not. If it's disabled, the API will still work but it won't collect anything. This is aimed at simplifying the stats collector usage: you should spend no more than one line of code for collecting stats in your spider, Scrapy extension, or whatever code you're using the Stats Collector from.
Another feature of the Stats Collector is that it's very efficient (when enabled) and extremely efficient (almost unnoticeable) when disabled.
See :ref:`topics-stats-reference` below for the stats that Scrapy sets.
System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "ref".Common Stats Collector uses
Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats` attribute. Here is an example of an extension that access stats:
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "attr".System Message: WARNING/2 (<stdin>, line 34)
Cannot analyze code. Pygments package not found.
.. code-block:: python
class ExtensionThatAccessStats:
def __init__(self, stats):
self.stats = stats
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
Set stat value:
System Message: WARNING/2 (<stdin>, line 48)
Cannot analyze code. Pygments package not found.
.. code-block:: python
stats.set_value("hostname", socket.gethostname())
Increment stat value:
System Message: WARNING/2 (<stdin>, line 54)
Cannot analyze code. Pygments package not found.
.. code-block:: python
stats.inc_value("custom_count")
Set stat value only if greater than previous:
System Message: WARNING/2 (<stdin>, line 60)
Cannot analyze code. Pygments package not found.
.. code-block:: python
stats.max_value("max_items_scraped", value)
Set stat value only if lower than previous:
System Message: WARNING/2 (<stdin>, line 66)
Cannot analyze code. Pygments package not found.
.. code-block:: python
stats.min_value("min_free_memory_percent", value)
Get stat value:
System Message: WARNING/2 (<stdin>, line 72)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> stats.get_value("custom_count")
1
Get all stats:
System Message: WARNING/2 (<stdin>, line 79)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> stats.get_stats()
{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
Available Stats Collectors
System Message: ERROR/3 (<stdin>, line 89)
Unknown directive type "currentmodule".
.. currentmodule:: scrapy.statscollectors
Besides the basic :class:`StatsCollector` there are other Stats Collectors available in Scrapy which extend the basic Stats Collector. You can select which Stats Collector to use through the :setting:`STATS_CLASS` setting. The default Stats Collector used is the :class:`MemoryStatsCollector`.
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "class".MemoryStatsCollector
System Message: ERROR/3 (<stdin>, line 99)
Unknown directive type "autoclass".
.. autoclass:: MemoryStatsCollector :members:
DummyStatsCollector
System Message: ERROR/3 (<stdin>, line 105)
Unknown directive type "autoclass".
.. autoclass:: DummyStatsCollector
Built-in stats reference
Scrapy sets the following :ref:`stats <topics-stats>`. Components other than those built into Scrapy may set additional stats; see their documentation.
System Message: ERROR/3 (<stdin>, line 112); backlink
Unknown interpreted text role "ref".Stat keys that contain a {placeholder} below stand for a family of stats, one per actual value of the placeholder.
Note
Most stats are set by a specific :ref:`component <topics-components>`, and are only present if that component is enabled and its code path is reached. A stat that is missing from :meth:`~scrapy.statscollectors.StatsCollector.get_stats` output is equivalent to a counter of 0.
System Message: ERROR/3 (<stdin>, line 118); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 118); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 124)
Unknown directive type "stat".
.. stat:: downloader/exception_count
- downloader/exception_count
Number of exceptions raised while downloading requests.
Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 129); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 131)
Unknown directive type "stat".
.. stat:: downloader/exception_type_count/{exception_type}
- downloader/exception_type_count/{exception_type}
Number of exceptions raised while downloading requests, per exception type, where {exception_type} is the import path of the exception class, e.g. twisted.internet.error.DNSLookupError.
Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 138); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 140)
Unknown directive type "stat".
.. stat:: downloader/request_bytes
- downloader/request_bytes
Total size, in bytes, of the requests sent, counting the request line, the headers and the body. As with :stat:`downloader/request_count`, requests served from the cache are also counted.
System Message: ERROR/3 (<stdin>, line 143); backlink
Unknown interpreted text role "stat".It is an approximation, reconstructed from each :class:`~scrapy.Request` object instead of measured on the wire, so it does not account for the actual bytes that the :ref:`download handler <topics-download-handlers>` sends, e.g. transport-level overhead.
System Message: ERROR/3 (<stdin>, line 147); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 147); backlink
Unknown interpreted text role "ref".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 152); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 154)
Unknown directive type "stat".
.. stat:: downloader/request_count
- downloader/request_count
Number of requests sent.
Requests that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache are also counted, even though they are never sent, because it handles requests after :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 159); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 159); backlink
Unknown interpreted text role "class".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 164); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 166)
Unknown directive type "stat".
.. stat:: downloader/request_method_count/{method}
- downloader/request_method_count/{method}
Number of requests sent, per HTTP method, e.g. GET or POST. As with :stat:`downloader/request_count`, requests served from the cache are also counted.
System Message: ERROR/3 (<stdin>, line 169); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 173); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 175)
Unknown directive type "stat".
.. stat:: downloader/response_bytes
- downloader/response_bytes
Total size, in bytes, of the responses received, counting the status line, the headers and the body. It covers the same responses as :stat:`downloader/response_count`.
System Message: ERROR/3 (<stdin>, line 178); backlink
Unknown interpreted text role "stat".The body is counted as received, i.e. still compressed for responses that used Content-Encoding, because :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` handles responses before :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware` decompresses them. See :stat:`httpcompression/response_bytes` for decompressed sizes.
System Message: ERROR/3 (<stdin>, line 182); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 182); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 182); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 190); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 192)
Unknown directive type "stat".
.. stat:: downloader/response_count
- downloader/response_count
Number of responses received.
It counts responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache, even though they do not come from the network, and responses that a downloader middleware consumes before they reach your spider, e.g. redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` turns into new requests. Compare with :stat:`response_received_count`.
System Message: ERROR/3 (<stdin>, line 197); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 197); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 197); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 203); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 205)
Unknown directive type "stat".
.. stat:: downloader/response_status_count/{status_code}
- downloader/response_status_count/{status_code}
Number of responses received, per HTTP status code, e.g. 200 or 404. It covers the same responses as :stat:`downloader/response_count`.
System Message: ERROR/3 (<stdin>, line 208); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats`.
System Message: ERROR/3 (<stdin>, line 211); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 213)
Unknown directive type "stat".
.. stat:: dupefilter/filtered
- dupefilter/filtered
Number of requests dropped as duplicates.
Set by :class:`~scrapy.dupefilters.RFPDupeFilter`.
System Message: ERROR/3 (<stdin>, line 218); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 220)
Unknown directive type "stat".
.. stat:: elapsed_time_seconds
- elapsed_time_seconds
Time, as a :class:`float`, in seconds, between the :signal:`spider_opened` and the :signal:`spider_closed` signals.
System Message: ERROR/3 (<stdin>, line 223); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 223); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 223); backlink
Unknown interpreted text role "signal".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 226); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 228)
Unknown directive type "stat".
.. stat:: feedexport/failed_count/{storage}
- feedexport/failed_count/{storage}
Number of :ref:`feeds <topics-feed-exports>` that could not be stored, per :ref:`storage backend <topics-feed-storage-backends>`, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.
System Message: ERROR/3 (<stdin>, line 231); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 231); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 235)
Unknown directive type "stat".
.. stat:: feedexport/success_count/{storage}
- feedexport/success_count/{storage}
Number of :ref:`feeds <topics-feed-exports>` stored successfully, per :ref:`storage backend <topics-feed-storage-backends>`, where {storage} is the class name of the storage backend, e.g. FileFeedStorage.
System Message: ERROR/3 (<stdin>, line 238); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 238); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 242)
Unknown directive type "stat".
.. stat:: file_count
- file_count
Number of files handled by the :ref:`media pipelines <topics-media-pipeline>`.
System Message: ERROR/3 (<stdin>, line 245); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 248)
Unknown directive type "stat".
.. stat:: file_status_count/{status}
- file_status_count/{status}
Number of files handled by the :ref:`media pipelines <topics-media-pipeline>`, per status, where {status} is one of:
System Message: ERROR/3 (<stdin>, line 251); backlink
Unknown interpreted text role "ref".downloaded: the file was downloaded.
cached: the file came from the :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` cache.
System Message: ERROR/3 (<stdin>, line 256); backlink
Unknown interpreted text role "class".
uptodate: the file was already in the storage backend and had not :ref:`expired <file-expiration>`, so it was not downloaded again.
System Message: ERROR/3 (<stdin>, line 260); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 263)
Unknown directive type "stat".
.. stat:: finish_reason
- finish_reason
String indicating why the crawl finished. It matches the reason argument of the :signal:`spider_closed` signal.
System Message: ERROR/3 (<stdin>, line 266); backlink
Unknown interpreted text role "signal".Scrapy uses the following reasons:
cancelled: the spider was closed without a more specific reason, e.g. because :exc:`~scrapy.exceptions.CloseSpider` was raised without one.
System Message: ERROR/3 (<stdin>, line 271); backlink
Unknown interpreted text role "exc".
closespider_errorcount: see :setting:`CLOSESPIDER_ERRORCOUNT`.
System Message: ERROR/3 (<stdin>, line 275); backlink
Unknown interpreted text role "setting".
closespider_itemcount: see :setting:`CLOSESPIDER_ITEMCOUNT`.
System Message: ERROR/3 (<stdin>, line 277); backlink
Unknown interpreted text role "setting".
closespider_pagecount: see :setting:`CLOSESPIDER_PAGECOUNT`.
System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "setting".
closespider_pagecount_no_item: see :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`.
System Message: ERROR/3 (<stdin>, line 281); backlink
Unknown interpreted text role "setting".
closespider_timeout: see :setting:`CLOSESPIDER_TIMEOUT`.
System Message: ERROR/3 (<stdin>, line 284); backlink
Unknown interpreted text role "setting".
closespider_timeout_no_item: see :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`.
System Message: ERROR/3 (<stdin>, line 286); backlink
Unknown interpreted text role "setting".
finished: the spider became idle with no pending requests, i.e. it finished normally.
memusage_exceeded: see :setting:`MEMUSAGE_LIMIT_MB`.
System Message: ERROR/3 (<stdin>, line 292); backlink
Unknown interpreted text role "setting".
shutdown: the crawl was interrupted, e.g. by a system signal such as SIGINT (:kbd:`Ctrl-C`).
System Message: ERROR/3 (<stdin>, line 294); backlink
Unknown interpreted text role "kbd".
Third-party components and your own code may use any other reason, e.g. by raising :exc:`~scrapy.exceptions.CloseSpider` with it.
System Message: ERROR/3 (<stdin>, line 297); backlink
Unknown interpreted text role "exc".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 300); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 302)
Unknown directive type "stat".
.. stat:: finish_time
- finish_time
Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when the :signal:`spider_closed` signal was sent.
System Message: ERROR/3 (<stdin>, line 305); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 305); backlink
Unknown interpreted text role "signal".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 308); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 310)
Unknown directive type "stat".
.. stat:: httpcache/errorrecovery
- httpcache/errorrecovery
Number of times that a stale cached response was used because downloading a fresh response raised an exception.
Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 316); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 318)
Unknown directive type "stat".
.. stat:: httpcache/firsthand
- httpcache/firsthand
Number of responses that were downloaded without a matching cache entry to validate against, i.e. responses for requests counted in :stat:`httpcache/miss`.
System Message: ERROR/3 (<stdin>, line 321); backlink
Unknown interpreted text role "stat".It is lower than :stat:`httpcache/miss` when some of those requests yield no response, either because they are dropped (see :stat:`httpcache/ignore`) or because their download fails.
System Message: ERROR/3 (<stdin>, line 325); backlink
Unknown interpreted text role "stat".System Message: ERROR/3 (<stdin>, line 325); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 329); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 331)
Unknown directive type "stat".
.. stat:: httpcache/hit
- httpcache/hit
Number of requests served from the cache.
Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 336); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 338)
Unknown directive type "stat".
.. stat:: httpcache/ignore
- httpcache/ignore
Number of requests dropped because they were not in the cache and :setting:`HTTPCACHE_IGNORE_MISSING` is True.
System Message: ERROR/3 (<stdin>, line 341); backlink
Unknown interpreted text role "setting".Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 344); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 346)
Unknown directive type "stat".
.. stat:: httpcache/invalidate
- httpcache/invalidate
Number of times that a cached response failed validation and was replaced with a freshly downloaded response.
Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 352); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 354)
Unknown directive type "stat".
.. stat:: httpcache/miss
- httpcache/miss
Number of requests for which no cache entry could be read, either because there was none or because reading it failed, in which case the request is also counted in :stat:`httpcache/retrieve_error`. Those requests are downloaded (see :stat:`httpcache/firsthand`), or dropped if :setting:`HTTPCACHE_IGNORE_MISSING` is True (see :stat:`httpcache/ignore`).
System Message: ERROR/3 (<stdin>, line 357); backlink
Unknown interpreted text role "stat".System Message: ERROR/3 (<stdin>, line 357); backlink
Unknown interpreted text role "stat".System Message: ERROR/3 (<stdin>, line 357); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 357); backlink
Unknown interpreted text role "stat".Requests with a stale cache entry are not counted here; see :stat:`httpcache/revalidate` and :stat:`httpcache/invalidate`.
System Message: ERROR/3 (<stdin>, line 364); backlink
Unknown interpreted text role "stat".System Message: ERROR/3 (<stdin>, line 364); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 367); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 369)
Unknown directive type "stat".
.. stat:: httpcache/retrieve_error
- httpcache/retrieve_error
Number of cache entries that could not be read, and hence were treated as cache misses. Those requests are also counted in :stat:`httpcache/miss`.
System Message: ERROR/3 (<stdin>, line 372); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 375); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 377)
Unknown directive type "stat".
.. stat:: httpcache/revalidate
- httpcache/revalidate
Number of times that a cached response was successfully validated against the target server, and hence used instead of the fresh response.
Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 383); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 385)
Unknown directive type "stat".
.. stat:: httpcache/store
- httpcache/store
Number of responses stored in the cache.
Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 390); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 392)
Unknown directive type "stat".
.. stat:: httpcache/uncacheable
- httpcache/uncacheable
Number of responses not stored in the cache because the :setting:`HTTPCACHE_POLICY` did not allow it.
System Message: ERROR/3 (<stdin>, line 395); backlink
Unknown interpreted text role "setting".Every response considered for caching is counted either here or in :stat:`httpcache/store`, so httpcache/store + httpcache/uncacheable equals httpcache/firsthand + httpcache/invalidate.
System Message: ERROR/3 (<stdin>, line 398); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.
System Message: ERROR/3 (<stdin>, line 402); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 404)
Unknown directive type "stat".
.. stat:: httpcompression/response_bytes
- httpcompression/response_bytes
Total size, in bytes, of decompressed response bodies, counting only the body and only responses that were actually decompressed. Compare with :stat:`downloader/response_bytes`.
System Message: ERROR/3 (<stdin>, line 407); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.
System Message: ERROR/3 (<stdin>, line 411); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 414)
Unknown directive type "stat".
.. stat:: httpcompression/response_count
- httpcompression/response_count
Number of decompressed responses.
Set by :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`.
System Message: ERROR/3 (<stdin>, line 419); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 422)
Unknown directive type "stat".
.. stat:: httperror/response_ignored_count
- httperror/response_ignored_count
Number of responses dropped because of their HTTP status code.
Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.
System Message: ERROR/3 (<stdin>, line 427); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 429)
Unknown directive type "stat".
.. stat:: httperror/response_ignored_status_count/{status_code}
- httperror/response_ignored_status_count/{status_code}
Number of responses dropped because of their HTTP status code, per HTTP status code, e.g. 404.
Set by :class:`~scrapy.spidermiddlewares.httperror.HttpErrorMiddleware`.
System Message: ERROR/3 (<stdin>, line 435); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 437)
Unknown directive type "stat".
.. stat:: item_dropped_count
- item_dropped_count
Number of items dropped by an :ref:`item pipeline <topics-item-pipeline>`, i.e. number of times that the :signal:`item_dropped` signal was sent.
System Message: ERROR/3 (<stdin>, line 440); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 440); backlink
Unknown interpreted text role "signal".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 444); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 446)
Unknown directive type "stat".
.. stat:: item_dropped_reasons_count/{exception}
- item_dropped_reasons_count/{exception}
Number of items dropped, per exception, where {exception} is the class name of the exception that caused the item to be dropped.
Only :exc:`~scrapy.exceptions.DropItem` and its subclasses drop items, and each one is counted under its own class name, e.g. item_dropped_reasons_count/DropItem for :exc:`~scrapy.exceptions.DropItem` itself and item_dropped_reasons_count/MyDropItem for a MyDropItem subclass of it. Any other exception raised by an :ref:`item pipeline <topics-item-pipeline>` triggers the :signal:`item_error` signal instead of :signal:`item_dropped`, and is not counted here or in :stat:`item_dropped_count`.
System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "exc".System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "exc".System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "stat".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 462); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 464)
Unknown directive type "stat".
.. stat:: item_scraped_count
- item_scraped_count
Number of items that passed all :ref:`item pipelines <topics-item-pipeline>`, i.e. number of times that the :signal:`item_scraped` signal was sent.
System Message: ERROR/3 (<stdin>, line 467); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 467); backlink
Unknown interpreted text role "signal".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 471); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 473)
Unknown directive type "stat".
.. stat:: items_per_minute
- items_per_minute
Average number of items scraped per minute during the crawl.
It is None if the crawl took less than a minute.
Set by :class:`~scrapy.extensions.logstats.LogStats`.
System Message: ERROR/3 (<stdin>, line 480); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 482)
Unknown directive type "stat".
.. stat:: log_count/{level}
- log_count/{level}
Number of log messages, per logging level name, e.g. INFO or WARNING.
Only messages that the :setting:`LOG_LEVEL` setting allows are counted.
System Message: ERROR/3 (<stdin>, line 488); backlink
Unknown interpreted text role "setting".Set by :class:`~scrapy.extensions.logcount.LogCount`.
System Message: ERROR/3 (<stdin>, line 490); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 492)
Unknown directive type "stat".
.. stat:: memdebug/gc_garbage_count
- memdebug/gc_garbage_count
Number of objects in :data:`gc.garbage` when the spider is closed.
System Message: ERROR/3 (<stdin>, line 495); backlink
Unknown interpreted text role "data".Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires :setting:`MEMDEBUG_ENABLED` to be True.
System Message: ERROR/3 (<stdin>, line 497); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 497); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 500)
Unknown directive type "stat".
.. stat:: memdebug/live_refs/{cls}
- memdebug/live_refs/{cls}
Number of live objects of class {cls} when the spider is closed, as reported by :ref:`trackref <topics-leaks-trackrefs>`, e.g. memdebug/live_refs/HtmlResponse.
System Message: ERROR/3 (<stdin>, line 503); backlink
Unknown interpreted text role "ref".Only set for classes with at least 1 live object.
Set by :class:`~scrapy.extensions.memdebug.MemoryDebugger`, which requires :setting:`MEMDEBUG_ENABLED` to be True.
System Message: ERROR/3 (<stdin>, line 509); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 509); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 512)
Unknown directive type "stat".
.. stat:: memusage/limit_reached
- memusage/limit_reached
1 if memory usage exceeded :setting:`MEMUSAGE_LIMIT_MB`, which also stops the crawl.
System Message: ERROR/3 (<stdin>, line 515); backlink
Unknown interpreted text role "setting".Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
System Message: ERROR/3 (<stdin>, line 518); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 520)
Unknown directive type "stat".
.. stat:: memusage/max
- memusage/max
Maximum peak memory usage, in bytes, observed during the crawl.
Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
System Message: ERROR/3 (<stdin>, line 525); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 527)
Unknown directive type "stat".
.. stat:: memusage/startup
- memusage/startup
Peak memory usage, in bytes, when the engine started.
Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
System Message: ERROR/3 (<stdin>, line 532); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 534)
Unknown directive type "stat".
.. stat:: memusage/warning_reached
- memusage/warning_reached
1 if memory usage exceeded :setting:`MEMUSAGE_WARNING_MB`.
System Message: ERROR/3 (<stdin>, line 537); backlink
Unknown interpreted text role "setting".Set by :class:`~scrapy.extensions.memusage.MemoryUsage`.
System Message: ERROR/3 (<stdin>, line 539); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 541)
Unknown directive type "stat".
.. stat:: offsite/domains
- offsite/domains
Number of distinct domains for which at least 1 request was dropped for being offsite.
Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.
System Message: ERROR/3 (<stdin>, line 547); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 549)
Unknown directive type "stat".
.. stat:: offsite/filtered
- offsite/filtered
Number of requests dropped for being offsite.
Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.
System Message: ERROR/3 (<stdin>, line 554); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 556)
Unknown directive type "stat".
.. stat:: request_depth_count/{depth}
- request_depth_count/{depth}
Number of requests scheduled at depth {depth}, e.g. request_depth_count/2.
Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`, which requires :setting:`DEPTH_STATS_VERBOSE` to be True for this stat.
System Message: ERROR/3 (<stdin>, line 562); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 562); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 565)
Unknown directive type "stat".
.. stat:: request_depth_max
- request_depth_max
Maximum depth reached.
Set by :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware`.
System Message: ERROR/3 (<stdin>, line 570); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 572)
Unknown directive type "stat".
.. stat:: response_received_count
- response_received_count
Number of responses received, i.e. number of times that the :signal:`response_received` signal was sent.
System Message: ERROR/3 (<stdin>, line 575); backlink
Unknown interpreted text role "signal".Unlike :stat:`downloader/response_count`, it does not count responses that a downloader middleware consumes before they reach the engine, e.g. redirect responses that :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` turns into new requests. Both count responses that :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` serves from the cache.
System Message: ERROR/3 (<stdin>, line 578); backlink
Unknown interpreted text role "stat".System Message: ERROR/3 (<stdin>, line 578); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 578); backlink
Unknown interpreted text role "class".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 584); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 586)
Unknown directive type "stat".
.. stat:: responses_per_minute
- responses_per_minute
Average number of responses received per minute during the crawl.
It is None if the crawl took less than a minute.
Set by :class:`~scrapy.extensions.logstats.LogStats`.
System Message: ERROR/3 (<stdin>, line 593); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 595)
Unknown directive type "stat".
.. stat:: retry/count
- retry/count
Number of requests retried.
Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
System Message: ERROR/3 (<stdin>, line 600); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 600); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 603)
Unknown directive type "stat".
.. stat:: retry/max_reached
- retry/max_reached
Number of requests that were not retried because they had already been retried :setting:`RETRY_TIMES` times.
System Message: ERROR/3 (<stdin>, line 606); backlink
Unknown interpreted text role "setting".Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
System Message: ERROR/3 (<stdin>, line 609); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 609); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 612)
Unknown directive type "stat".
.. stat:: retry/reason_count/{reason}
- retry/reason_count/{reason}
Number of requests retried, per reason, e.g. retry/reason_count/twisted.internet.error.TimeoutError or retry/reason_count/504 Gateway Time-out.
Set by :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, which :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses.
System Message: ERROR/3 (<stdin>, line 619); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 619); backlink
Unknown interpreted text role "class".
Note
Code calling :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` may pass a custom stats_base_key, in which case retry is replaced with that key in the 3 stats above.
System Message: ERROR/3 (<stdin>, line 622); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 627)
Unknown directive type "stat".
.. stat:: robotstxt/exception_count/{exception_type}
- robotstxt/exception_count/{exception_type}
Number of exceptions raised while downloading robots.txt files, per exception type, where {exception_type} is the string representation of the exception class, e.g. <class 'twisted.internet.error.DNSLookupError'>.
Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
System Message: ERROR/3 (<stdin>, line 635); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 638)
Unknown directive type "stat".
.. stat:: robotstxt/forbidden
- robotstxt/forbidden
Number of requests dropped for being disallowed by robots.txt.
Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
System Message: ERROR/3 (<stdin>, line 643); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 646)
Unknown directive type "stat".
.. stat:: robotstxt/request_count
- robotstxt/request_count
Number of robots.txt files requested, i.e. 1 per network location for which at least 1 request was sent.
Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
System Message: ERROR/3 (<stdin>, line 652); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 655)
Unknown directive type "stat".
.. stat:: robotstxt/response_count
- robotstxt/response_count
Number of robots.txt responses received.
Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
System Message: ERROR/3 (<stdin>, line 660); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 663)
Unknown directive type "stat".
.. stat:: robotstxt/response_status_count/{status_code}
- robotstxt/response_status_count/{status_code}
Number of robots.txt responses received, per HTTP status code, e.g. 404.
Set by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`.
System Message: ERROR/3 (<stdin>, line 669); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 672)
Unknown directive type "stat".
.. stat:: scheduler/dequeued
- scheduler/dequeued
Number of requests read from the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 675); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 677)
Unknown directive type "stat".
.. stat:: scheduler/dequeued/disk
- scheduler/dequeued/disk
Number of requests read from the disk queue of the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 680); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 683)
Unknown directive type "stat".
.. stat:: scheduler/dequeued/memory
- scheduler/dequeued/memory
Number of requests read from the memory queue of the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 686); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 689)
Unknown directive type "stat".
.. stat:: scheduler/enqueued
- scheduler/enqueued
Number of requests stored into the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 692); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 694)
Unknown directive type "stat".
.. stat:: scheduler/enqueued/disk
- scheduler/enqueued/disk
Number of requests stored into the disk queue of the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 697); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 700)
Unknown directive type "stat".
.. stat:: scheduler/enqueued/memory
- scheduler/enqueued/memory
Number of requests stored into the memory queue of the :ref:`scheduler <topics-scheduler>`.
System Message: ERROR/3 (<stdin>, line 703); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 706)
Unknown directive type "stat".
.. stat:: scheduler/unserializable
- scheduler/unserializable
Number of requests that could not be stored into the disk queue of the :ref:`scheduler <topics-scheduler>` because they could not be :ref:`serialized <request-serialization>`, and hence were stored into the memory queue instead.
System Message: ERROR/3 (<stdin>, line 709); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 709); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 714)
Unknown directive type "stat".
.. stat:: spider_exceptions/count
- spider_exceptions/count
Number of unhandled exceptions raised by spider callbacks.
Set by the :ref:`scraper <topics-architecture>`.
System Message: ERROR/3 (<stdin>, line 719); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 721)
Unknown directive type "stat".
.. stat:: spider_exceptions/{exception}
- spider_exceptions/{exception}
Number of unhandled exceptions raised by spider callbacks, per exception, where {exception} is the class name of the exception, e.g. spider_exceptions/ValueError.
Set by the :ref:`scraper <topics-architecture>`.
System Message: ERROR/3 (<stdin>, line 728); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 730)
Unknown directive type "stat".
.. stat:: start_time
- start_time
Timezone-aware :class:`~datetime.datetime` object, in UTC, indicating when the :signal:`spider_opened` signal was sent.
System Message: ERROR/3 (<stdin>, line 733); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 733); backlink
Unknown interpreted text role "signal".Set by :class:`~scrapy.extensions.corestats.CoreStats`.
System Message: ERROR/3 (<stdin>, line 736); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 738)
Unknown directive type "stat".
.. stat:: urllength/request_ignored_count
- urllength/request_ignored_count
Number of requests dropped for having a URL longer than :setting:`URLLENGTH_LIMIT`.
System Message: ERROR/3 (<stdin>, line 741); backlink
Unknown interpreted text role "setting".Set by :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`.
System Message: ERROR/3 (<stdin>, line 744); backlink
Unknown interpreted text role "class".