RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method

This commit is contained in:
Pablo Hoffman 2009-11-14 20:28:59 -02:00
parent c4c6e7c8cd
commit aeab5370cb
18 changed files with 246 additions and 249 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

160
docs/topics/stats.rst
View File

@ -8,9 +8,9 @@ Overview
======== ========
Scrapy provides a convenient service for collecting stats in the form of Scrapy provides a convenient service for collecting stats in the form of
key/values, both globally and per spider/domain. It's called the Stats key/values, both globally and per spider. It's called the Stats Collector, and
Collector, and it's a singleton which can be imported and used quickly, as it's a singleton which can be imported and used quickly, as illustrated by the
illustrated by the examples in the :ref:`topics-stats-usecases` section below. examples in the :ref:`topics-stats-usecases` section below.
The stats collection is enabled by default but can be disabled through the The stats collection is enabled by default but can be disabled through the
:setting:`STATS_ENABLED` setting. :setting:`STATS_ENABLED` setting.
@ -26,10 +26,10 @@ using the Stats Collector from.
Another feature of the Stats Collector is that it's very efficient (when Another feature of the Stats Collector is that it's very efficient (when
enabled) and extremely efficient (almost unnoticeable) when disabled. enabled) and extremely efficient (almost unnoticeable) when disabled.
The Stats Collector keeps one stats table per open spider/domain and one global The Stats Collector keeps one stats table per open spider and one global stats
stats table. You can't set or get stats from a closed domain, but the table. You can't set or get stats from a closed spider, but the spider-specific
domain-specific stats table is automatically opened when the spider is opened, stats table is automatically opened when the spider is opened, and closed when
and closed when the spider is closed. the spider is closed.
.. _topics-stats-usecases: .. _topics-stats-usecases:
@ -61,36 +61,38 @@ Get global stat value::
>>> stats.get_value('spiders_crawled') >>> stats.get_value('spiders_crawled')
8 8
Get all global stats from a given domain:: Get all global stats (ie. not particular to any spider)::
>>> stats.get_stats() >>> stats.get_stats()
{'hostname': 'localhost', 'spiders_crawled': 8} {'hostname': 'localhost', 'spiders_crawled': 8}
Set domain/spider specific stat value (domains must be opened first, but this Set spider specific stat value (spider stats must be opened first, but this
task is handled automatically by the Scrapy engine):: task is handled automatically by the Scrapy engine)::
stats.set_value('start_time', datetime.now(), domain='example.com') stats.set_value('start_time', datetime.now(), spider=some_spider)
Increment domain-specific stat value:: Where ``some_spider`` is a :class:`~scrapy.spider.BaseSpider` object.
stats.inc_value('pages_crawled', domain='example.com') Increment spider-specific stat value::
Set domain-specific stat value only if greater than previous:: stats.inc_value('pages_crawled', spider=some_spider)
stats.max_value('max_items_scraped', value, domain='example.com') Set spider-specific stat value only if greater than previous::
Set domain-specific stat value only if lower than previous:: stats.max_value('max_items_scraped', value, spider=some_spider)
stats.min_value('min_free_memory_percent', value, domain='example.com') Set spider-specific stat value only if lower than previous::
Get domain-specific stat value:: stats.min_value('min_free_memory_percent', value, spider=some_spider)
>>> stats.get_value('pages_crawled', domain='example.com') Get spider-specific stat value::
>>> stats.get_value('pages_crawled', spider=some_spider)
1238 1238
Get all stats from a given domain:: Get all stats from a given spider::
>>> stats.get_stats('pages_crawled', domain='example.com') >>> stats.get_stats('pages_crawled', spider=some_spider)
{'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} {'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
.. _topics-stats-ref: .. _topics-stats-ref:
@ -108,74 +110,79 @@ class (which they all inherit from).
.. class:: StatsCollector .. class:: StatsCollector
.. method:: get_value(key, default=None, domain=None) .. method:: get_value(key, default=None, spider=None)
Return the value for the given stats key or default if it doesn't exist. Return the value for the given stats key or default if it doesn't exist.
If domain is ``None`` the global stats table is consulted, other the If spider is ``None`` the global stats table is consulted, otherwise the
domain specific one is. If the domain is not yet opened a ``KeyError`` spider specific one is. If the spider is not yet opened a ``KeyError``
exception is raised. exception is raised.
.. method:: get_stats(domain=None) .. method:: get_stats(spider=None)
Get all stats from the given domain/spider (if domain is given) or all Get all stats from the given spider (if spider is given) or all global
global stats otherwise, as a dict. If domain is not opened ``KeyError`` stats otherwise, as a dict. If spider is not opened ``KeyError`` is
is raied. raied.
.. method:: set_value(key, value, domain=None) .. method:: set_value(key, value, spider=None)
Set the given value for the given stats key on the global stats (if Set the given value for the given stats key on the global stats (if
domain is not given) or the domain-specific stats (if domain is given), spider is not given) or the spider-specific stats (if spider is given),
which must be opened or a ``KeyError`` will be raised. which must be opened or a ``KeyError`` will be raised.
.. method:: set_stats(stats, domain=None) .. method:: set_stats(stats, spider=None)
Set the given stats (as a dict) for the given domain. If the domain is Set the given stats (as a dict) for the given spider. If the spider is
not opened a ``KeyError`` will be raised. not opened a ``KeyError`` will be raised.
.. method:: inc_value(key, count=1, start=0, domain=None) .. method:: inc_value(key, count=1, start=0, spider=None)
Increment the value of the given stats key, by the given count, Increment the value of the given stats key, by the given count,
assuming the start value given (when it's not set). If domain is not assuming the start value given (when it's not set). If spider is not
given the global stats table is used, otherwise the domain-specific given the global stats table is used, otherwise the spider-specific
stats table is used, which must be opened or a ``KeyError`` will be stats table is used, which must be opened or a ``KeyError`` will be
raised. raised.
.. method:: max_value(key, value, domain=None) .. method:: max_value(key, value, spider=None)
Set the given value for the given key only if current value for the Set the given value for the given key only if current value for the
same key is lower than value. If there is no current value for the same key is lower than value. If there is no current value for the
given key, the value is always set. If domain is not given the global given key, the value is always set. If spider is not given the global
stats table is used, otherwise the domain-specific stats table is used, stats table is used, otherwise the spider-specific stats table is used,
which must be opened or a KeyError will be raised. which must be opened or a KeyError will be raised.
.. method:: min_value(key, value, domain=None) .. method:: min_value(key, value, spider=None)
Set the given value for the given key only if current value for the Set the given value for the given key only if current value for the
same key is greater than value. If there is no current value for the same key is greater than value. If there is no current value for the
given key, the value is always set. If domain is not given the global given key, the value is always set. If spider is not given the global
stats table is used, otherwise the domain-specific stats table is used, stats table is used, otherwise the spider-specific stats table is used,
which must be opened or a KeyError will be raised. which must be opened or a KeyError will be raised.
.. method:: clear_stats(domain=None) .. method:: clear_stats(spider=None)
Clear all global stats (if domain is not given) or all domain-specific Clear all global stats (if spider is not given) or all spider-specific
stats if domain is given, in which case it must be opened or a stats if spider is given, in which case it must be opened or a
``KeyError`` will be raised. ``KeyError`` will be raised.
.. method:: list_domains() .. method:: iter_spider_stats()
Return a list of all opened domains. Return a iterator over ``(spider, spider_stats)`` for each open spider
currently tracked by the stats collector, where ``spider_stats`` is the
dict containing all spider-specific stats.
.. method:: open_domain(domain) Global stats are not included in the iterator. If you want to get
those, use :meth:`get_stats` method.
Open the given domain for stats collection. This method must be called .. method:: open_spider(spider)
prior to working with any stats specific to that domain, but this task
Open the given spider for stats collection. This method must be called
prior to working with any stats specific to that spider, but this task
is handled automatically by the Scrapy engine. is handled automatically by the Scrapy engine.
.. method:: close_domain(domain) .. method:: close_spider(spider)
Close the given domain. After this is called, no more specific stats Close the given spider. After this is called, no more specific stats
for this domain can be accessed. This method is called automatically on for this spider can be accessed. This method is called automatically on
the :signal:`spider_closed` signal. the :signal:`spider_closed` signal.
Available Stats Collectors Available Stats Collectors
@ -196,15 +203,16 @@ MemoryStatsCollector
.. class:: MemoryStatsCollector .. class:: MemoryStatsCollector
A simple stats collector that keeps the stats of the last scraping run (for A simple stats collector that keeps the stats of the last scraping run (for
each domain) in memory, which can be accessed through the ``domain_stats`` each spider) in memory, after they're closed. The stats can be accessed
attribute through the :attr:`domain_stats` attribute, which is a dict keyed by spider
domain name.
This is the default Stats Collector used in Scrapy. This is the default Stats Collector used in Scrapy.
.. attribute:: domain_stats .. attribute:: domain_stats
A dict of dicts (keyed by domain) containing the stats of the last A dict of dicts (keyed by spider domain name) containing the stats of
scraping run for each domain. the last scraping run for each domain.
DummyStatsCollector DummyStatsCollector
------------------- -------------------
@ -283,41 +291,41 @@ functionality:
.. module:: scrapy.stats.signals .. module:: scrapy.stats.signals
:synopsis: Stats Collector signals :synopsis: Stats Collector signals
.. signal:: stats_domain_opened .. signal:: stats_spider_opened
.. function:: stats_domain_opened(domain) .. function:: stats_spider_opened(spider)
Sent right after the stats domain is opened. You can use this signal to add Sent right after the stats spider is opened. You can use this signal to add
startup stats for domain (example: start time). startup stats for spider (example: start time).
:param domain: the stats domain just opened :param spider: the stats spider just opened
:type domain: str :type spider: str
.. signal:: stats_domain_closing .. signal:: stats_spider_closing
.. function:: stats_domain_closing(domain, reason) .. function:: stats_spider_closing(spider, reason)
Sent just before the stats domain is closed. You can use this signal to add Sent just before the stats spider is closed. You can use this signal to add
some closing stats (example: finish time). some closing stats (example: finish time).
:param domain: the stats domain about to be closed :param spider: the stats spider about to be closed
:type domain: str :type spider: str
:param reason: the reason why the domain is being closed. See :param reason: the reason why the spider is being closed. See
:signal:`spider_closed` signal for more info. :signal:`spider_closed` signal for more info.
:type reason: str :type reason: str
.. signal:: stats_domain_closed .. signal:: stats_spider_closed
.. function:: stats_domain_closed(domain, reason, domain_stats) .. function:: stats_spider_closed(spider, reason, spider_stats)
Sent right after the stats domain is closed. You can use this signal to Sent right after the stats spider is closed. You can use this signal to
collect resources, but not to add any more stats as the stats domain has collect resources, but not to add any more stats as the stats spider has
already been close (use :signal:`stats_domain_closing` for that instead). already been close (use :signal:`stats_spider_closing` for that instead).
:param domain: the stats domain just closed :param spider: the stats spider just closed
:type domain: str :type spider: str
:param reason: the reason why the domain was closed. See :param reason: the reason why the spider was closed. See
:signal:`spider_closed` signal for more info. :signal:`spider_closed` signal for more info.
:type reason: str :type reason: str
:param domain_stats: the stats of the domain just closed. :param spider_stats: the stats of the spider just closed.
:type reason: dict :type reason: dict

30
scrapy/contrib/corestats.py
View File

@ -10,7 +10,7 @@ from scrapy.xlib.pydispatch import dispatcher
from scrapy.core import signals from scrapy.core import signals
from scrapy.stats import stats from scrapy.stats import stats
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing from scrapy.stats.signals import stats_spider_opened, stats_spider_closing
from scrapy.conf import settings from scrapy.conf import settings
class CoreStats(object): class CoreStats(object):
@ -22,32 +22,32 @@ class CoreStats(object):
stats.set_value('envinfo/logfile', settings['LOG_FILE']) stats.set_value('envinfo/logfile', settings['LOG_FILE'])
stats.set_value('envinfo/pid', os.getpid()) stats.set_value('envinfo/pid', os.getpid())
dispatcher.connect(self.stats_domain_opened, signal=stats_domain_opened) dispatcher.connect(self.stats_spider_opened, signal=stats_spider_opened)
dispatcher.connect(self.stats_domain_closing, signal=stats_domain_closing) dispatcher.connect(self.stats_spider_closing, signal=stats_spider_closing)
dispatcher.connect(self.item_scraped, signal=signals.item_scraped) dispatcher.connect(self.item_scraped, signal=signals.item_scraped)
dispatcher.connect(self.item_passed, signal=signals.item_passed) dispatcher.connect(self.item_passed, signal=signals.item_passed)
dispatcher.connect(self.item_dropped, signal=signals.item_dropped) dispatcher.connect(self.item_dropped, signal=signals.item_dropped)
def stats_domain_opened(self, domain): def stats_spider_opened(self, spider):
stats.set_value('start_time', datetime.datetime.utcnow(), domain=domain) stats.set_value('start_time', datetime.datetime.utcnow(), spider=spider)
stats.set_value('envinfo/host', stats.get_value('envinfo/host'), domain=domain) stats.set_value('envinfo/host', stats.get_value('envinfo/host'), spider=spider)
stats.inc_value('domain_count/opened') stats.inc_value('spider_count/opened')
def stats_domain_closing(self, domain, reason): def stats_spider_closing(self, spider, reason):
stats.set_value('finish_time', datetime.datetime.utcnow(), domain=domain) stats.set_value('finish_time', datetime.datetime.utcnow(), spider=spider)
stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, domain=domain) stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, spider=spider)
stats.inc_value('domain_count/%s' % reason, domain=domain) stats.inc_value('spider_count/%s' % reason, spider=spider)
def item_scraped(self, item, spider): def item_scraped(self, item, spider):
stats.inc_value('item_scraped_count', domain=spider.domain_name) stats.inc_value('item_scraped_count', spider=spider)
stats.inc_value('item_scraped_count') stats.inc_value('item_scraped_count')
def item_passed(self, item, spider): def item_passed(self, item, spider):
stats.inc_value('item_passed_count', domain=spider.domain_name) stats.inc_value('item_passed_count', spider=spider)
stats.inc_value('item_passed_count') stats.inc_value('item_passed_count')
def item_dropped(self, item, spider, exception): def item_dropped(self, item, spider, exception):
reason = exception.__class__.__name__ reason = exception.__class__.__name__
stats.inc_value('item_dropped_count', domain=spider.domain_name) stats.inc_value('item_dropped_count', spider=spider)
stats.inc_value('item_dropped_reasons_count/%s' % reason, domain=spider.domain_name) stats.inc_value('item_dropped_reasons_count/%s' % reason, spider=spider)
stats.inc_value('item_dropped_count') stats.inc_value('item_dropped_count')

23
scrapy/contrib/downloadermiddleware/stats.py
View File

@ -5,37 +5,30 @@ from scrapy.stats import stats
from scrapy.conf import settings from scrapy.conf import settings
class DownloaderStats(object): class DownloaderStats(object):
"""DownloaderStats store stats of all requests, responses and
exceptions that pass through it.
To use this middleware you must enable the DOWNLOADER_STATS setting.
"""
def __init__(self): def __init__(self):
if not settings.getbool('DOWNLOADER_STATS'): if not settings.getbool('DOWNLOADER_STATS'):
raise NotConfigured raise NotConfigured
def process_request(self, request, spider): def process_request(self, request, spider):
domain = spider.domain_name
stats.inc_value('downloader/request_count') stats.inc_value('downloader/request_count')
stats.inc_value('downloader/request_count', domain=domain) stats.inc_value('downloader/request_count', spider=spider)
stats.inc_value('downloader/request_method_count/%s' % request.method, domain=domain) stats.inc_value('downloader/request_method_count/%s' % request.method, spider=spider)
reqlen = len(request_httprepr(request)) reqlen = len(request_httprepr(request))
stats.inc_value('downloader/request_bytes', reqlen, domain=domain) stats.inc_value('downloader/request_bytes', reqlen, spider=spider)
stats.inc_value('downloader/request_bytes', reqlen) stats.inc_value('downloader/request_bytes', reqlen)
def process_response(self, request, response, spider): def process_response(self, request, response, spider):
domain = spider.domain_name
stats.inc_value('downloader/response_count') stats.inc_value('downloader/response_count')
stats.inc_value('downloader/response_count', domain=domain) stats.inc_value('downloader/response_count', spider=spider)
stats.inc_value('downloader/response_status_count/%s' % response.status, domain=domain) stats.inc_value('downloader/response_status_count/%s' % response.status, spider=spider)
reslen = len(response_httprepr(response)) reslen = len(response_httprepr(response))
stats.inc_value('downloader/response_bytes', reslen, domain=domain) stats.inc_value('downloader/response_bytes', reslen, spider=spider)
stats.inc_value('downloader/response_bytes', reslen) stats.inc_value('downloader/response_bytes', reslen)
return response return response
def process_exception(self, request, exception, spider): def process_exception(self, request, exception, spider):
ex_class = "%s.%s" % (exception.__class__.__module__, exception.__class__.__name__) ex_class = "%s.%s" % (exception.__class__.__module__, exception.__class__.__name__)
stats.inc_value('downloader/exception_count') stats.inc_value('downloader/exception_count')
stats.inc_value('downloader/exception_count', domain=spider.domain_name) stats.inc_value('downloader/exception_count', spider=spider)
stats.inc_value('downloader/exception_type_count/%s' % ex_class, domain=spider.domain_name) stats.inc_value('downloader/exception_type_count/%s' % ex_class, spider=spider)

10
scrapy/contrib/itemsampler.py
View File

@ -53,11 +53,11 @@ class ItemSamplerPipeline(object):
dispatcher.connect(self.engine_stopped, signal=signals.engine_stopped) dispatcher.connect(self.engine_stopped, signal=signals.engine_stopped)
def process_item(self, spider, item): def process_item(self, spider, item):
sampled = stats.get_value("items_sampled", 0, domain=spider.domain_name) sampled = stats.get_value("items_sampled", 0, spider=spider)
if sampled < items_per_spider: if sampled < items_per_spider:
self.items[item.guid] = item self.items[item.guid] = item
sampled += 1 sampled += 1
stats.set_value("items_sampled", sampled, domain=spider.domain_name) stats.set_value("items_sampled", sampled, spider=spider)
log.msg("Sampled %s" % item, spider=spider, level=log.INFO) log.msg("Sampled %s" % item, spider=spider, level=log.INFO)
if close_spider and sampled == items_per_spider: if close_spider and sampled == items_per_spider:
scrapyengine.close_spider(spider) scrapyengine.close_spider(spider)
@ -71,7 +71,7 @@ class ItemSamplerPipeline(object):
level=log.WARNING) level=log.WARNING)
def spider_closed(self, spider, reason): def spider_closed(self, spider, reason):
if reason == 'finished' and not stats.get_value("items_sampled", domain=spider.domain_name): if reason == 'finished' and not stats.get_value("items_sampled", spider=spider):
self.empty_domains.add(spider.domain_name) self.empty_domains.add(spider.domain_name)
self.spiders_count += 1 self.spiders_count += 1
log.msg("Sampled %d domains so far (%d empty)" % (self.spiders_count, \ log.msg("Sampled %d domains so far (%d empty)" % (self.spiders_count, \
@ -87,7 +87,7 @@ class ItemSamplerMiddleware(object):
raise NotConfigured raise NotConfigured
def process_spider_input(self, response, spider): def process_spider_input(self, response, spider):
if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider: if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
return [] return []
elif max_response_size and max_response_size > len(response_httprepr(response)): elif max_response_size and max_response_size > len(response_httprepr(response)):
return [] return []
@ -100,7 +100,7 @@ class ItemSamplerMiddleware(object):
else: else:
items.append(r) items.append(r)
if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider: if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
return [] return []
else: else:
# TODO: this needs some revision, as keeping only the first item # TODO: this needs some revision, as keeping only the first item

10
scrapy/contrib/pipeline/images.py
View File

@ -219,7 +219,7 @@ class ImagesPipeline(MediaPipeline):
msg = 'Image (%s): Downloaded image from %s referred in <%s>' % \ msg = 'Image (%s): Downloaded image from %s referred in <%s>' % \
(status, request, referer) (status, request, referer)
log.msg(msg, level=log.DEBUG, spider=info.spider) log.msg(msg, level=log.DEBUG, spider=info.spider)
self.inc_stats(info.spider.domain_name, status) self.inc_stats(info.spider, status)
try: try:
key = self.image_key(request.url) key = self.image_key(request.url)
@ -258,7 +258,7 @@ class ImagesPipeline(MediaPipeline):
referer = request.headers.get('Referer') referer = request.headers.get('Referer')
log.msg('Image (uptodate): Downloaded %s from <%s> referred in <%s>' % \ log.msg('Image (uptodate): Downloaded %s from <%s> referred in <%s>' % \
(self.MEDIA_NAME, request.url, referer), level=log.DEBUG, spider=info.spider) (self.MEDIA_NAME, request.url, referer), level=log.DEBUG, spider=info.spider)
self.inc_stats(info.spider.domain_name, 'uptodate') self.inc_stats(info.spider, 'uptodate')
checksum = result.get('checksum', None) checksum = result.get('checksum', None)
return {'url': request.url, 'path': key, 'checksum': checksum} return {'url': request.url, 'path': key, 'checksum': checksum}
@ -295,9 +295,9 @@ class ImagesPipeline(MediaPipeline):
thumb_image, thumb_buf = self.convert_image(image, size) thumb_image, thumb_buf = self.convert_image(image, size)
yield thumb_key, thumb_image, thumb_buf yield thumb_key, thumb_image, thumb_buf
def inc_stats(self, domain, status): def inc_stats(self, spider, status):
stats.inc_value('image_count', domain=domain) stats.inc_value('image_count', spider=spider)
stats.inc_value('image_status_count/%s' % status, domain=domain) stats.inc_value('image_status_count/%s' % status, spider=spider)
def convert_image(self, image, size=None): def convert_image(self, image, size=None):
if image.mode != 'RGB': if image.mode != 'RGB':

9
scrapy/contrib/spidermiddleware/depth.py
View File

@ -18,7 +18,6 @@ class DepthMiddleware(object):
stats.set_value('envinfo/request_depth_limit', self.maxdepth) stats.set_value('envinfo/request_depth_limit', self.maxdepth)
def process_spider_output(self, response, result, spider): def process_spider_output(self, response, result, spider):
domain = spider.domain_name
def _filter(request): def _filter(request):
if isinstance(request, Request): if isinstance(request, Request):
depth = response.request.meta['depth'] + 1 depth = response.request.meta['depth'] + 1
@ -28,14 +27,14 @@ class DepthMiddleware(object):
level=log.DEBUG, spider=spider) level=log.DEBUG, spider=spider)
return False return False
elif self.stats: elif self.stats:
stats.inc_value('request_depth_count/%s' % depth, domain=domain) stats.inc_value('request_depth_count/%s' % depth, spider=spider)
if depth > stats.get_value('request_depth_max', 0, domain=domain): if depth > stats.get_value('request_depth_max', 0, spider=spider):
stats.set_value('request_depth_max', depth, domain=domain) stats.set_value('request_depth_max', depth, spider=spider)
return True return True
# base case (depth=0) # base case (depth=0)
if self.stats and 'depth' not in response.request.meta: if self.stats and 'depth' not in response.request.meta:
response.request.meta['depth'] = 0 response.request.meta['depth'] = 0
stats.inc_value('request_depth_count/0', domain=domain) stats.inc_value('request_depth_count/0', spider=spider)
return (r for r in result or () if _filter(r)) return (r for r in result or () if _filter(r))

12
scrapy/contrib/statsmailer.py
View File

@ -1,5 +1,5 @@
""" """
StatsMailer extension sends an email when a domain finishes scraping. StatsMailer extension sends an email when a spider finishes scraping.
Use STATSMAILER_RCPTS setting to enable and give the recipient mail address Use STATSMAILER_RCPTS setting to enable and give the recipient mail address
""" """
@ -17,12 +17,12 @@ class StatsMailer(object):
self.recipients = settings.getlist("STATSMAILER_RCPTS") self.recipients = settings.getlist("STATSMAILER_RCPTS")
if not self.recipients: if not self.recipients:
raise NotConfigured raise NotConfigured
dispatcher.connect(self.stats_domain_closed, signal=signals.stats_domain_closed) dispatcher.connect(self.stats_spider_closed, signal=signals.stats_spider_closed)
def stats_domain_closed(self, domain, domain_stats): def stats_spider_closed(self, spider, spider_stats):
mail = MailSender() mail = MailSender()
body = "Global stats\n\n" body = "Global stats\n\n"
body += "\n".join("%-50s : %s" % i for i in stats.get_stats().items()) body += "\n".join("%-50s : %s" % i for i in stats.get_stats().items())
body += "\n\n%s stats\n\n" % domain body += "\n\n%s stats\n\n" % spider.domain_name
body += "\n".join("%-50s : %s" % i for i in domain_stats.items()) body += "\n".join("%-50s : %s" % i for i in spider_stats.items())
mail.send(self.recipients, "Scrapy stats for: %s" % domain, body) mail.send(self.recipients, "Scrapy stats for: %s" % spider.domain_name, body)

6
scrapy/contrib/webconsole/stats.py
View File

@ -22,9 +22,9 @@ class StatsDump(object):
s = banner(self) s = banner(self)
s += "<h3>Global stats</h3>\n" s += "<h3>Global stats</h3>\n"
s += stats_html_table(stats.get_stats()) s += stats_html_table(stats.get_stats())
for domain in stats.list_domains(): for spider, spider_stats in stats.iter_spider_stats():
s += "<h3>%s</h3>\n" % domain s += "<h3>%s</h3>\n" % spider.domain_name
s += stats_html_table(stats.get_stats(domain)) s += stats_html_table(spider_stats)
s += "</body>\n" s += "</body>\n"
s += "</html>\n" s += "</html>\n"

16
scrapy/contrib_exp/spiderprofiler.py
View File

@ -45,20 +45,18 @@ class SpiderProfiler(object):
r = function(*args, **kwargs) r = function(*args, **kwargs)
mafter = self._memusage() mafter = self._memusage()
ct = time() - tbefore ct = time() - tbefore
domain = spider.domain_name tcc = stats.get_value('profiling/total_callback_time', 0, spider=spider)
tcc = stats.get_value('profiling/total_callback_time', 0, domain=domain) sct = stats.get_value('profiling/slowest_callback_time', 0, spider=spider)
sct = stats.get_value('profiling/slowest_callback_time', 0, domain=domain) stats.set_value('profiling/total_callback_time', tcc+ct, spider=spider)
stats.set_value('profiling/total_callback_time' % spider.domain_name, \
tcc+ct, domain=domain)
if ct > sct: if ct > sct:
stats.set_value('profiling/slowest_callback_time', ct, domain=domain) stats.set_value('profiling/slowest_callback_time', ct, spider=spider)
stats.set_value('profiling/slowest_callback_name', function.__name__, \ stats.set_value('profiling/slowest_callback_name', function.__name__, \
domain=domain) spider=spider)
stats.set_value('profiling/slowest_callback_url', args[0].url, \ stats.set_value('profiling/slowest_callback_url', args[0].url, \
domain=domain) spider=spider)
if self._memusage: if self._memusage:
stats.inc_value('profiling/total_mem_allocated_in_callbacks', \ stats.inc_value('profiling/total_mem_allocated_in_callbacks', \
count=mafter-mbefore, domain=domain) count=mafter-mbefore, spider=spider)
return r return r
return new_callback return new_callback

4
scrapy/core/engine.py
View File

@ -244,7 +244,7 @@ class ExecutionEngine(object):
self.downloader.open_spider(spider) self.downloader.open_spider(spider)
self.scraper.open_spider(spider) self.scraper.open_spider(spider)
stats.open_domain(spider.domain_name) stats.open_spider(spider)
send_catch_log(signals.spider_opened, sender=self.__class__, spider=spider) send_catch_log(signals.spider_opened, sender=self.__class__, spider=spider)
@ -305,7 +305,7 @@ class ExecutionEngine(object):
reason = self.closing.pop(spider, 'finished') reason = self.closing.pop(spider, 'finished')
send_catch_log(signal=signals.spider_closed, sender=self.__class__, \ send_catch_log(signal=signals.spider_closed, sender=self.__class__, \
spider=spider, reason=reason) spider=spider, reason=reason)
stats.close_domain(spider.domain_name, reason=reason) stats.close_spider(spider, reason=reason)
dfd = defer.maybeDeferred(spiders.close_spider, spider) dfd = defer.maybeDeferred(spiders.close_spider, spider)
dfd.addBoth(log.msg, "Spider closed (%s)" % reason, spider=spider) dfd.addBoth(log.msg, "Spider closed (%s)" % reason, spider=spider)
reactor.callLater(0, self._mainloop) reactor.callLater(0, self._mainloop)

14
scrapy/core/scraper.py
View File

@ -87,17 +87,17 @@ class Scraper(object):
def enqueue_scrape(self, response, request, spider): def enqueue_scrape(self, response, request, spider):
site = self.sites[spider] site = self.sites[spider]
dfd = site.add_response_request(response, request) dfd = site.add_response_request(response, request)
# FIXME: this can't be called here because the stats domain may be # FIXME: this can't be called here because the stats spider may be
# already closed # already closed
#stats.max_value('scraper/max_active_size', site.active_size, \ #stats.max_value('scraper/max_active_size', site.active_size, \
# domain=spider.domain_name) # spider=spider)
def finish_scraping(_): def finish_scraping(_):
site.finish_response(response) site.finish_response(response)
self._scrape_next(spider, site) self._scrape_next(spider, site)
return _ return _
dfd.addBoth(finish_scraping) dfd.addBoth(finish_scraping)
dfd.addErrback(log.err, 'Scraper bug processing %s' % request, \ dfd.addErrback(log.err, 'Scraper bug processing %s' % request, \
domain=spider.domain_name) spider=spider)
self._scrape_next(spider, site) self._scrape_next(spider, site)
return dfd return dfd
@ -138,7 +138,7 @@ class Scraper(object):
(request.url, referer, _failure) (request.url, referer, _failure)
log.msg(msg, log.ERROR, spider=spider) log.msg(msg, log.ERROR, spider=spider)
stats.inc_value("spider_exceptions/%s" % _failure.value.__class__.__name__, \ stats.inc_value("spider_exceptions/%s" % _failure.value.__class__.__name__, \
domain=spider.domain_name) spider=spider)
def handle_spider_output(self, result, request, response, spider): def handle_spider_output(self, result, request, response, spider):
if not result: if not result:
@ -152,7 +152,6 @@ class Scraper(object):
from the given spider from the given spider
""" """
# TODO: keep closing state internally instead of checking engine # TODO: keep closing state internally instead of checking engine
domain = spider.domain_name
if spider in self.engine.closing: if spider in self.engine.closing:
return return
elif isinstance(output, Request): elif isinstance(output, Request):
@ -165,10 +164,10 @@ class Scraper(object):
send_catch_log(signal=signals.item_scraped, sender=self.__class__, \ send_catch_log(signal=signals.item_scraped, sender=self.__class__, \
item=output, spider=spider, response=response) item=output, spider=spider, response=response)
self.sites[spider].itemproc_size += 1 self.sites[spider].itemproc_size += 1
# FIXME: this can't be called here because the stats domain may be # FIXME: this can't be called here because the stats spider may be
# already closed # already closed
#stats.max_value('scraper/max_itemproc_size', \ #stats.max_value('scraper/max_itemproc_size', \
# self.sites[domain].itemproc_size, domain=domain) # self.sites[spider].itemproc_size, spider=spider)
dfd = self.itemproc.process_item(output, spider) dfd = self.itemproc.process_item(output, spider)
dfd.addBoth(self._itemproc_finished, output, spider) dfd.addBoth(self._itemproc_finished, output, spider)
return dfd return dfd
@ -195,7 +194,6 @@ class Scraper(object):
def _itemproc_finished(self, output, item, spider): def _itemproc_finished(self, output, item, spider):
"""ItemProcessor finished for the given ``item`` and returned ``output`` """ItemProcessor finished for the given ``item`` and returned ``output``
""" """
domain = spider.domain_name
self.sites[spider].itemproc_size -= 1 self.sites[spider].itemproc_size -= 1
if isinstance(output, Failure): if isinstance(output, Failure):
ex = output.value ex = output.value

85
scrapy/stats/collector/__init__.py
View File

@ -5,8 +5,8 @@ import pprint
from scrapy.xlib.pydispatch import dispatcher from scrapy.xlib.pydispatch import dispatcher
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \ from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
stats_domain_closed stats_spider_closed
from scrapy.utils.signal import send_catch_log from scrapy.utils.signal import send_catch_log
from scrapy.core import signals from scrapy.core import signals
from scrapy import log from scrapy import log
@ -19,57 +19,57 @@ class StatsCollector(object):
self._stats = {None: {}} # None is for global stats self._stats = {None: {}} # None is for global stats
dispatcher.connect(self._engine_stopped, signal=signals.engine_stopped) dispatcher.connect(self._engine_stopped, signal=signals.engine_stopped)
def get_value(self, key, default=None, domain=None): def get_value(self, key, default=None, spider=None):
return self._stats[domain].get(key, default) return self._stats[spider].get(key, default)
def get_stats(self, domain=None): def get_stats(self, spider=None):
return self._stats[domain] return self._stats[spider]
def set_value(self, key, value, domain=None): def set_value(self, key, value, spider=None):
self._stats[domain][key] = value self._stats[spider][key] = value
def set_stats(self, stats, domain=None): def set_stats(self, stats, spider=None):
self._stats[domain] = stats self._stats[spider] = stats
def inc_value(self, key, count=1, start=0, domain=None): def inc_value(self, key, count=1, start=0, spider=None):
d = self._stats[domain] d = self._stats[spider]
d[key] = d.setdefault(key, start) + count d[key] = d.setdefault(key, start) + count
def max_value(self, key, value, domain=None): def max_value(self, key, value, spider=None):
d = self._stats[domain] d = self._stats[spider]
d[key] = max(d.setdefault(key, value), value) d[key] = max(d.setdefault(key, value), value)
def min_value(self, key, value, domain=None): def min_value(self, key, value, spider=None):
d = self._stats[domain] d = self._stats[spider]
d[key] = min(d.setdefault(key, value), value) d[key] = min(d.setdefault(key, value), value)
def clear_stats(self, domain=None): def clear_stats(self, spider=None):
self._stats[domain].clear() self._stats[spider].clear()
def list_domains(self): def iter_spider_stats(self):
return [d for d in self._stats.keys() if d is not None] return [x for x in self._stats.iteritems() if x[0]]
def open_domain(self, domain): def open_spider(self, spider):
self._stats[domain] = {} self._stats[spider] = {}
send_catch_log(stats_domain_opened, domain=domain) send_catch_log(stats_spider_opened, spider=spider)
def close_domain(self, domain, reason): def close_spider(self, spider, reason):
send_catch_log(stats_domain_closing, domain=domain, reason=reason) send_catch_log(stats_spider_closing, spider=spider, reason=reason)
stats = self._stats.pop(domain) stats = self._stats.pop(spider)
send_catch_log(stats_domain_closed, domain=domain, reason=reason, \ send_catch_log(stats_spider_closed, spider=spider, reason=reason, \
domain_stats=stats) spider_stats=stats)
if self._dump: if self._dump:
log.msg("Dumping domain stats:\n" + pprint.pformat(stats), \ log.msg("Dumping spider stats:\n" + pprint.pformat(stats), \
domain=domain) spider=spider)
self._persist_stats(stats, domain) self._persist_stats(stats, spider)
def _engine_stopped(self): def _engine_stopped(self):
stats = self.get_stats() stats = self.get_stats()
if self._dump: if self._dump:
log.msg("Dumping global stats:\n" + pprint.pformat(stats)) log.msg("Dumping global stats:\n" + pprint.pformat(stats))
self._persist_stats(stats, domain=None) self._persist_stats(stats, spider=None)
def _persist_stats(self, stats, domain=None): def _persist_stats(self, stats, spider=None):
pass pass
class MemoryStatsCollector(StatsCollector): class MemoryStatsCollector(StatsCollector):
@ -77,29 +77,30 @@ class MemoryStatsCollector(StatsCollector):
def __init__(self): def __init__(self):
super(MemoryStatsCollector, self).__init__() super(MemoryStatsCollector, self).__init__()
self.domain_stats = {} self.domain_stats = {}
def _persist_stats(self, stats, domain=None): def _persist_stats(self, stats, spider=None):
self.domain_stats[domain] = stats if spider is not None:
self.domain_stats[spider.domain_name] = stats
class DummyStatsCollector(StatsCollector): class DummyStatsCollector(StatsCollector):
def get_value(self, key, default=None, domain=None): def get_value(self, key, default=None, spider=None):
return default return default
def set_value(self, key, value, domain=None): def set_value(self, key, value, spider=None):
pass pass
def set_stats(self, stats, domain=None): def set_stats(self, stats, spider=None):
pass pass
def inc_value(self, key, count=1, start=0, domain=None): def inc_value(self, key, count=1, start=0, spider=None):
pass pass
def max_value(self, key, value, domain=None): def max_value(self, key, value, spider=None):
pass pass
def min_value(self, key, value, domain=None): def min_value(self, key, value, spider=None):
pass pass

6
scrapy/stats/collector/mysql.py
View File

@ -16,8 +16,8 @@ class MysqlStatsCollector(StatsCollector):
mysqluri = settings['STATS_MYSQL_URI'] mysqluri = settings['STATS_MYSQL_URI']
self._mysql_conn = mysql_connect(mysqluri, use_unicode=False) if mysqluri else None self._mysql_conn = mysql_connect(mysqluri, use_unicode=False) if mysqluri else None
def _persist_stats(self, stats, domain=None): def _persist_stats(self, stats, spider=None):
if domain is None: # only store domain-specific stats if spider is None: # only store spider-specific stats
return return
if self._mysql_conn is None: if self._mysql_conn is None:
return return
@ -27,5 +27,5 @@ class MysqlStatsCollector(StatsCollector):
c = self._mysql_conn.cursor() c = self._mysql_conn.cursor()
c.execute("INSERT INTO %s (domain,stored,data) VALUES (%%s,%%s,%%s)" % table, \ c.execute("INSERT INTO %s (domain,stored,data) VALUES (%%s,%%s,%%s)" % table, \
(domain, stored, datas)) (spider.domain_name, stored, datas))
self._mysql_conn.commit() self._mysql_conn.commit()

20
scrapy/stats/collector/simpledb.py
View File

@ -22,27 +22,27 @@ class SimpledbStatsCollector(StatsCollector):
self._async = settings.getbool('STATS_SDB_ASYNC') self._async = settings.getbool('STATS_SDB_ASYNC')
connect_sdb().create_domain(self._sdbdomain) connect_sdb().create_domain(self._sdbdomain)
def _persist_stats(self, stats, domain=None): def _persist_stats(self, stats, spider=None):
if domain is None: # only store domain-specific stats if spider is None: # only store spider-specific stats
return return
if not self._sdbdomain: if not self._sdbdomain:
return return
if self._async: if self._async:
dfd = threads.deferToThread(self._persist_to_sdb, domain, stats.copy()) dfd = threads.deferToThread(self._persist_to_sdb, spider, stats.copy())
dfd.addErrback(log.err, 'Error uploading stats to SimpleDB', \ dfd.addErrback(log.err, 'Error uploading stats to SimpleDB', \
domain=domain) spider=spider)
else: else:
self._persist_to_sdb(domain, stats) self._persist_to_sdb(spider, stats)
def _persist_to_sdb(self, domain, stats): def _persist_to_sdb(self, spider, stats):
ts = self._get_timestamp(domain).isoformat() ts = self._get_timestamp(spider).isoformat()
sdb_item_id = "%s_%s" % (domain, ts) sdb_item_id = "%s_%s" % (spider.domain_name, ts)
sdb_item = dict((k, self._to_sdb_value(v, k)) for k, v in stats.iteritems()) sdb_item = dict((k, self._to_sdb_value(v, k)) for k, v in stats.iteritems())
sdb_item['domain'] = domain sdb_item['domain'] = spider.domain_name
sdb_item['timestamp'] = self._to_sdb_value(ts) sdb_item['timestamp'] = self._to_sdb_value(ts)
connect_sdb().put_attributes(self._sdbdomain, sdb_item_id, sdb_item) connect_sdb().put_attributes(self._sdbdomain, sdb_item_id, sdb_item)
def _get_timestamp(self, domain): def _get_timestamp(self, spider):
return datetime.utcnow() return datetime.utcnow()
def _to_sdb_value(self, obj, key=None): def _to_sdb_value(self, obj, key=None):

6
scrapy/stats/signals.py
View File

@ -1,3 +1,3 @@
stats_domain_opened = object() stats_spider_opened = object()
stats_domain_closing = object() stats_spider_closing = object()
stats_domain_closed = object() stats_spider_closed = object()

14
scrapy/tests/test_downloadermiddleware_stats.py
View File

@ -1,6 +1,5 @@
from unittest import TestCase from unittest import TestCase
from scrapy.conf import settings
from scrapy.contrib.downloadermiddleware.stats import DownloaderStats from scrapy.contrib.downloadermiddleware.stats import DownloaderStats
from scrapy.http import Request, Response from scrapy.http import Request, Response
from scrapy.spider import BaseSpider from scrapy.spider import BaseSpider
@ -10,11 +9,10 @@ from scrapy.stats import stats
class TestDownloaderStats(TestCase): class TestDownloaderStats(TestCase):
def setUp(self): def setUp(self):
self.spider = BaseSpider() self.spider = BaseSpider('scrapytest.org')
self.spider.domain_name = 'scrapytest.org'
self.mw = DownloaderStats() self.mw = DownloaderStats()
stats.open_domain(self.spider.domain_name) stats.open_spider(self.spider)
self.req = Request('scrapytest.org') self.req = Request('scrapytest.org')
self.res = Response('scrapytest.org', status=400) self.res = Response('scrapytest.org', status=400)
@ -22,18 +20,18 @@ class TestDownloaderStats(TestCase):
def test_process_request(self): def test_process_request(self):
self.mw.process_request(self.req, self.spider) self.mw.process_request(self.req, self.spider)
self.assertEqual(stats.get_value('downloader/request_count', \ self.assertEqual(stats.get_value('downloader/request_count', \
domain=self.spider.domain_name), 1) spider=self.spider), 1)
def test_process_response(self): def test_process_response(self):
self.mw.process_response(self.req, self.res, self.spider) self.mw.process_response(self.req, self.res, self.spider)
self.assertEqual(stats.get_value('downloader/response_count', \ self.assertEqual(stats.get_value('downloader/response_count', \
domain=self.spider.domain_name), 1) spider=self.spider), 1)
def test_process_exception(self): def test_process_exception(self):
self.mw.process_exception(self.req, Exception(), self.spider) self.mw.process_exception(self.req, Exception(), self.spider)
self.assertEqual(stats.get_value('downloader/exception_count', \ self.assertEqual(stats.get_value('downloader/exception_count', \
domain=self.spider.domain_name), 1) spider=self.spider), 1)
def tearDown(self): def tearDown(self):
stats.close_domain(self.spider.domain_name, '') stats.close_spider(self.spider, '')

13
scrapy/tests/test_spidermiddleware_depth.py
View File

@ -14,10 +14,9 @@ class TestDepthMiddleware(TestCase):
settings.overrides['DEPTH_LIMIT'] = 1 settings.overrides['DEPTH_LIMIT'] = 1
settings.overrides['DEPTH_STATS'] = True settings.overrides['DEPTH_STATS'] = True
self.spider = BaseSpider() self.spider = BaseSpider('scrapytest.org')
self.spider.domain_name = 'scrapytest.org'
stats.open_domain(self.spider.domain_name) stats.open_spider(self.spider)
self.mw = DepthMiddleware() self.mw = DepthMiddleware()
self.assertEquals(stats.get_value('envinfo/request_depth_limit'), 1) self.assertEquals(stats.get_value('envinfo/request_depth_limit'), 1)
@ -31,8 +30,7 @@ class TestDepthMiddleware(TestCase):
out = list(self.mw.process_spider_output(resp, result, self.spider)) out = list(self.mw.process_spider_output(resp, result, self.spider))
self.assertEquals(out, result) self.assertEquals(out, result)
rdc = stats.get_value('request_depth_count/1', rdc = stats.get_value('request_depth_count/1', spider=self.spider)
domain=self.spider.domain_name)
self.assertEquals(rdc, 1) self.assertEquals(rdc, 1)
req.meta['depth'] = 1 req.meta['depth'] = 1
@ -40,8 +38,7 @@ class TestDepthMiddleware(TestCase):
out2 = list(self.mw.process_spider_output(resp, result, self.spider)) out2 = list(self.mw.process_spider_output(resp, result, self.spider))
self.assertEquals(out2, []) self.assertEquals(out2, [])
rdm = stats.get_value('request_depth_max', rdm = stats.get_value('request_depth_max', spider=self.spider)
domain=self.spider.domain_name)
self.assertEquals(rdm, 1) self.assertEquals(rdm, 1)
def tearDown(self): def tearDown(self):
@ -49,5 +46,5 @@ class TestDepthMiddleware(TestCase):
del settings.overrides['DEPTH_STATS'] del settings.overrides['DEPTH_STATS']
settings.disabled = True settings.disabled = True
stats.close_domain(self.spider.domain_name, '') stats.close_spider(self.spider, '')

57
scrapy/tests/test_stats.py
View File

@ -1,12 +1,16 @@
import unittest import unittest
from scrapy.spider import BaseSpider
from scrapy.xlib.pydispatch import dispatcher from scrapy.xlib.pydispatch import dispatcher
from scrapy.stats.collector import StatsCollector, DummyStatsCollector from scrapy.stats.collector import StatsCollector, DummyStatsCollector
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \ from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
stats_domain_closed stats_spider_closed
class StatsCollectorTest(unittest.TestCase): class StatsCollectorTest(unittest.TestCase):
def setUp(self):
self.spider = BaseSpider()
def test_collector(self): def test_collector(self):
stats = StatsCollector() stats = StatsCollector()
self.assertEqual(stats.get_stats(), {}) self.assertEqual(stats.get_stats(), {})
@ -43,44 +47,45 @@ class StatsCollectorTest(unittest.TestCase):
stats.inc_value('v1') stats.inc_value('v1')
stats.max_value('v2', 100) stats.max_value('v2', 100)
stats.min_value('v3', 100) stats.min_value('v3', 100)
stats.open_domain('a') stats.open_spider('a')
stats.set_value('test', 'value', domain='a') stats.set_value('test', 'value', spider=self.spider)
self.assertEqual(stats.get_stats(), {}) self.assertEqual(stats.get_stats(), {})
self.assertEqual(stats.get_stats('a'), {}) self.assertEqual(stats.get_stats('a'), {})
def test_signals(self): def test_signals(self):
signals_catched = set() signals_catched = set()
def domain_opened(domain): def spider_opened(spider):
assert domain == 'example.com' assert spider is self.spider
signals_catched.add(stats_domain_opened) signals_catched.add(stats_spider_opened)
def domain_closing(domain, reason): def spider_closing(spider, reason):
assert domain == 'example.com' assert spider is self.spider
assert reason == 'testing' assert reason == 'testing'
signals_catched.add(stats_domain_closing) signals_catched.add(stats_spider_closing)
def domain_closed(domain, reason, domain_stats): def spider_closed(spider, reason, spider_stats):
assert domain == 'example.com' assert spider is self.spider
assert reason == 'testing' assert reason == 'testing'
assert domain_stats == {'test': 1} assert spider_stats == {'test': 1}
signals_catched.add(stats_domain_closed) signals_catched.add(stats_spider_closed)
dispatcher.connect(domain_opened, signal=stats_domain_opened) dispatcher.connect(spider_opened, signal=stats_spider_opened)
dispatcher.connect(domain_closing, signal=stats_domain_closing) dispatcher.connect(spider_closing, signal=stats_spider_closing)
dispatcher.connect(domain_closed, signal=stats_domain_closed) dispatcher.connect(spider_closed, signal=stats_spider_closed)
stats = StatsCollector() stats = StatsCollector()
stats.open_domain('example.com') stats.open_spider(self.spider)
stats.set_value('test', 1, domain='example.com') stats.set_value('test', 1, spider=self.spider)
stats.close_domain('example.com', 'testing') self.assertEqual([(self.spider, {'test': 1})], list(stats.iter_spider_stats()))
assert stats_domain_opened in signals_catched stats.close_spider(self.spider, 'testing')
assert stats_domain_closing in signals_catched assert stats_spider_opened in signals_catched
assert stats_domain_closed in signals_catched assert stats_spider_closing in signals_catched
assert stats_spider_closed in signals_catched
dispatcher.disconnect(domain_opened, signal=stats_domain_opened) dispatcher.disconnect(spider_opened, signal=stats_spider_opened)
dispatcher.disconnect(domain_closing, signal=stats_domain_closing) dispatcher.disconnect(spider_closing, signal=stats_spider_closing)
dispatcher.disconnect(domain_closed, signal=stats_domain_closed) dispatcher.disconnect(spider_closed, signal=stats_spider_closed)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()