mirror of https://github.com/scrapy/scrapy.git
StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method
This commit is contained in:
Pablo Hoffman
parent
c4c6e7c8cd
commit
aeab5370cb
|
|
@ -8,9 +8,9 @@ Overview
|
|||
========
|
||||
|
||||
Scrapy provides a convenient service for collecting stats in the form of
|
||||
key/values, both globally and per spider/domain. It's called the Stats
|
||||
Collector, and it's a singleton which can be imported and used quickly, as
|
||||
illustrated by the examples in the :ref:`topics-stats-usecases` section below.
|
||||
key/values, both globally and per spider. It's called the Stats Collector, and
|
||||
it's a singleton which can be imported and used quickly, as illustrated by the
|
||||
examples in the :ref:`topics-stats-usecases` section below.
|
||||
|
||||
The stats collection is enabled by default but can be disabled through the
|
||||
:setting:`STATS_ENABLED` setting.
|
||||
|
|
@ -26,10 +26,10 @@ using the Stats Collector from.
|
|||
Another feature of the Stats Collector is that it's very efficient (when
|
||||
enabled) and extremely efficient (almost unnoticeable) when disabled.
|
||||
|
||||
The Stats Collector keeps one stats table per open spider/domain and one global
|
||||
stats table. You can't set or get stats from a closed domain, but the
|
||||
domain-specific stats table is automatically opened when the spider is opened,
|
||||
and closed when the spider is closed.
|
||||
The Stats Collector keeps one stats table per open spider and one global stats
|
||||
table. You can't set or get stats from a closed spider, but the spider-specific
|
||||
stats table is automatically opened when the spider is opened, and closed when
|
||||
the spider is closed.
|
||||
|
||||
.. _topics-stats-usecases:
|
||||
|
||||
|
|
@ -61,36 +61,38 @@ Get global stat value::
|
|||
>>> stats.get_value('spiders_crawled')
|
||||
8
|
||||
|
||||
Get all global stats from a given domain::
|
||||
Get all global stats (ie. not particular to any spider)::
|
||||
|
||||
>>> stats.get_stats()
|
||||
{'hostname': 'localhost', 'spiders_crawled': 8}
|
||||
|
||||
Set domain/spider specific stat value (domains must be opened first, but this
|
||||
Set spider specific stat value (spider stats must be opened first, but this
|
||||
task is handled automatically by the Scrapy engine)::
|
||||
|
||||
stats.set_value('start_time', datetime.now(), domain='example.com')
|
||||
stats.set_value('start_time', datetime.now(), spider=some_spider)
|
||||
|
||||
Increment domain-specific stat value::
|
||||
Where ``some_spider`` is a :class:`~scrapy.spider.BaseSpider` object.
|
||||
|
||||
stats.inc_value('pages_crawled', domain='example.com')
|
||||
Increment spider-specific stat value::
|
||||
|
||||
Set domain-specific stat value only if greater than previous::
|
||||
stats.inc_value('pages_crawled', spider=some_spider)
|
||||
|
||||
stats.max_value('max_items_scraped', value, domain='example.com')
|
||||
Set spider-specific stat value only if greater than previous::
|
||||
|
||||
Set domain-specific stat value only if lower than previous::
|
||||
stats.max_value('max_items_scraped', value, spider=some_spider)
|
||||
|
||||
stats.min_value('min_free_memory_percent', value, domain='example.com')
|
||||
Set spider-specific stat value only if lower than previous::
|
||||
|
||||
Get domain-specific stat value::
|
||||
stats.min_value('min_free_memory_percent', value, spider=some_spider)
|
||||
|
||||
>>> stats.get_value('pages_crawled', domain='example.com')
|
||||
Get spider-specific stat value::
|
||||
|
||||
>>> stats.get_value('pages_crawled', spider=some_spider)
|
||||
1238
|
||||
|
||||
Get all stats from a given domain::
|
||||
Get all stats from a given spider::
|
||||
|
||||
>>> stats.get_stats('pages_crawled', domain='example.com')
|
||||
>>> stats.get_stats('pages_crawled', spider=some_spider)
|
||||
{'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
|
||||
|
||||
.. _topics-stats-ref:
|
||||
|
|
@ -108,74 +110,79 @@ class (which they all inherit from).
|
|||
|
||||
.. class:: StatsCollector
|
||||
|
||||
.. method:: get_value(key, default=None, domain=None)
|
||||
.. method:: get_value(key, default=None, spider=None)
|
||||
|
||||
Return the value for the given stats key or default if it doesn't exist.
|
||||
If domain is ``None`` the global stats table is consulted, other the
|
||||
domain specific one is. If the domain is not yet opened a ``KeyError``
|
||||
If spider is ``None`` the global stats table is consulted, otherwise the
|
||||
spider specific one is. If the spider is not yet opened a ``KeyError``
|
||||
exception is raised.
|
||||
|
||||
.. method:: get_stats(domain=None)
|
||||
.. method:: get_stats(spider=None)
|
||||
|
||||
Get all stats from the given domain/spider (if domain is given) or all
|
||||
global stats otherwise, as a dict. If domain is not opened ``KeyError``
|
||||
is raied.
|
||||
Get all stats from the given spider (if spider is given) or all global
|
||||
stats otherwise, as a dict. If spider is not opened ``KeyError`` is
|
||||
raied.
|
||||
|
||||
.. method:: set_value(key, value, domain=None)
|
||||
.. method:: set_value(key, value, spider=None)
|
||||
|
||||
Set the given value for the given stats key on the global stats (if
|
||||
domain is not given) or the domain-specific stats (if domain is given),
|
||||
spider is not given) or the spider-specific stats (if spider is given),
|
||||
which must be opened or a ``KeyError`` will be raised.
|
||||
|
||||
.. method:: set_stats(stats, domain=None)
|
||||
.. method:: set_stats(stats, spider=None)
|
||||
|
||||
Set the given stats (as a dict) for the given domain. If the domain is
|
||||
Set the given stats (as a dict) for the given spider. If the spider is
|
||||
not opened a ``KeyError`` will be raised.
|
||||
|
||||
.. method:: inc_value(key, count=1, start=0, domain=None)
|
||||
.. method:: inc_value(key, count=1, start=0, spider=None)
|
||||
|
||||
Increment the value of the given stats key, by the given count,
|
||||
assuming the start value given (when it's not set). If domain is not
|
||||
given the global stats table is used, otherwise the domain-specific
|
||||
assuming the start value given (when it's not set). If spider is not
|
||||
given the global stats table is used, otherwise the spider-specific
|
||||
stats table is used, which must be opened or a ``KeyError`` will be
|
||||
raised.
|
||||
|
||||
.. method:: max_value(key, value, domain=None)
|
||||
.. method:: max_value(key, value, spider=None)
|
||||
|
||||
Set the given value for the given key only if current value for the
|
||||
same key is lower than value. If there is no current value for the
|
||||
given key, the value is always set. If domain is not given the global
|
||||
stats table is used, otherwise the domain-specific stats table is used,
|
||||
given key, the value is always set. If spider is not given the global
|
||||
stats table is used, otherwise the spider-specific stats table is used,
|
||||
which must be opened or a KeyError will be raised.
|
||||
|
||||
.. method:: min_value(key, value, domain=None)
|
||||
.. method:: min_value(key, value, spider=None)
|
||||
|
||||
Set the given value for the given key only if current value for the
|
||||
same key is greater than value. If there is no current value for the
|
||||
given key, the value is always set. If domain is not given the global
|
||||
stats table is used, otherwise the domain-specific stats table is used,
|
||||
given key, the value is always set. If spider is not given the global
|
||||
stats table is used, otherwise the spider-specific stats table is used,
|
||||
which must be opened or a KeyError will be raised.
|
||||
|
||||
.. method:: clear_stats(domain=None)
|
||||
.. method:: clear_stats(spider=None)
|
||||
|
||||
Clear all global stats (if domain is not given) or all domain-specific
|
||||
stats if domain is given, in which case it must be opened or a
|
||||
Clear all global stats (if spider is not given) or all spider-specific
|
||||
stats if spider is given, in which case it must be opened or a
|
||||
``KeyError`` will be raised.
|
||||
|
||||
.. method:: list_domains()
|
||||
.. method:: iter_spider_stats()
|
||||
|
||||
Return a list of all opened domains.
|
||||
Return a iterator over ``(spider, spider_stats)`` for each open spider
|
||||
currently tracked by the stats collector, where ``spider_stats`` is the
|
||||
dict containing all spider-specific stats.
|
||||
|
||||
.. method:: open_domain(domain)
|
||||
Global stats are not included in the iterator. If you want to get
|
||||
those, use :meth:`get_stats` method.
|
||||
|
||||
Open the given domain for stats collection. This method must be called
|
||||
prior to working with any stats specific to that domain, but this task
|
||||
.. method:: open_spider(spider)
|
||||
|
||||
Open the given spider for stats collection. This method must be called
|
||||
prior to working with any stats specific to that spider, but this task
|
||||
is handled automatically by the Scrapy engine.
|
||||
|
||||
.. method:: close_domain(domain)
|
||||
.. method:: close_spider(spider)
|
||||
|
||||
Close the given domain. After this is called, no more specific stats
|
||||
for this domain can be accessed. This method is called automatically on
|
||||
Close the given spider. After this is called, no more specific stats
|
||||
for this spider can be accessed. This method is called automatically on
|
||||
the :signal:`spider_closed` signal.
|
||||
|
||||
Available Stats Collectors
|
||||
|
|
@ -196,15 +203,16 @@ MemoryStatsCollector
|
|||
.. class:: MemoryStatsCollector
|
||||
|
||||
A simple stats collector that keeps the stats of the last scraping run (for
|
||||
each domain) in memory, which can be accessed through the ``domain_stats``
|
||||
attribute
|
||||
each spider) in memory, after they're closed. The stats can be accessed
|
||||
through the :attr:`domain_stats` attribute, which is a dict keyed by spider
|
||||
domain name.
|
||||
|
||||
This is the default Stats Collector used in Scrapy.
|
||||
|
||||
.. attribute:: domain_stats
|
||||
|
||||
A dict of dicts (keyed by domain) containing the stats of the last
|
||||
scraping run for each domain.
|
||||
A dict of dicts (keyed by spider domain name) containing the stats of
|
||||
the last scraping run for each domain.
|
||||
|
||||
DummyStatsCollector
|
||||
-------------------
|
||||
|
|
@ -283,41 +291,41 @@ functionality:
|
|||
.. module:: scrapy.stats.signals
|
||||
:synopsis: Stats Collector signals
|
||||
|
||||
.. signal:: stats_domain_opened
|
||||
.. function:: stats_domain_opened(domain)
|
||||
.. signal:: stats_spider_opened
|
||||
.. function:: stats_spider_opened(spider)
|
||||
|
||||
Sent right after the stats domain is opened. You can use this signal to add
|
||||
startup stats for domain (example: start time).
|
||||
Sent right after the stats spider is opened. You can use this signal to add
|
||||
startup stats for spider (example: start time).
|
||||
|
||||
:param domain: the stats domain just opened
|
||||
:type domain: str
|
||||
:param spider: the stats spider just opened
|
||||
:type spider: str
|
||||
|
||||
.. signal:: stats_domain_closing
|
||||
.. function:: stats_domain_closing(domain, reason)
|
||||
.. signal:: stats_spider_closing
|
||||
.. function:: stats_spider_closing(spider, reason)
|
||||
|
||||
Sent just before the stats domain is closed. You can use this signal to add
|
||||
Sent just before the stats spider is closed. You can use this signal to add
|
||||
some closing stats (example: finish time).
|
||||
|
||||
:param domain: the stats domain about to be closed
|
||||
:type domain: str
|
||||
:param spider: the stats spider about to be closed
|
||||
:type spider: str
|
||||
|
||||
:param reason: the reason why the domain is being closed. See
|
||||
:param reason: the reason why the spider is being closed. See
|
||||
:signal:`spider_closed` signal for more info.
|
||||
:type reason: str
|
||||
|
||||
.. signal:: stats_domain_closed
|
||||
.. function:: stats_domain_closed(domain, reason, domain_stats)
|
||||
.. signal:: stats_spider_closed
|
||||
.. function:: stats_spider_closed(spider, reason, spider_stats)
|
||||
|
||||
Sent right after the stats domain is closed. You can use this signal to
|
||||
collect resources, but not to add any more stats as the stats domain has
|
||||
already been close (use :signal:`stats_domain_closing` for that instead).
|
||||
Sent right after the stats spider is closed. You can use this signal to
|
||||
collect resources, but not to add any more stats as the stats spider has
|
||||
already been close (use :signal:`stats_spider_closing` for that instead).
|
||||
|
||||
:param domain: the stats domain just closed
|
||||
:type domain: str
|
||||
:param spider: the stats spider just closed
|
||||
:type spider: str
|
||||
|
||||
:param reason: the reason why the domain was closed. See
|
||||
:param reason: the reason why the spider was closed. See
|
||||
:signal:`spider_closed` signal for more info.
|
||||
:type reason: str
|
||||
|
||||
:param domain_stats: the stats of the domain just closed.
|
||||
:param spider_stats: the stats of the spider just closed.
|
||||
:type reason: dict
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from scrapy.xlib.pydispatch import dispatcher
|
|||
|
||||
from scrapy.core import signals
|
||||
from scrapy.stats import stats
|
||||
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing
|
||||
from scrapy.stats.signals import stats_spider_opened, stats_spider_closing
|
||||
from scrapy.conf import settings
|
||||
|
||||
class CoreStats(object):
|
||||
|
|
@ -22,32 +22,32 @@ class CoreStats(object):
|
|||
stats.set_value('envinfo/logfile', settings['LOG_FILE'])
|
||||
stats.set_value('envinfo/pid', os.getpid())
|
||||
|
||||
dispatcher.connect(self.stats_domain_opened, signal=stats_domain_opened)
|
||||
dispatcher.connect(self.stats_domain_closing, signal=stats_domain_closing)
|
||||
dispatcher.connect(self.stats_spider_opened, signal=stats_spider_opened)
|
||||
dispatcher.connect(self.stats_spider_closing, signal=stats_spider_closing)
|
||||
dispatcher.connect(self.item_scraped, signal=signals.item_scraped)
|
||||
dispatcher.connect(self.item_passed, signal=signals.item_passed)
|
||||
dispatcher.connect(self.item_dropped, signal=signals.item_dropped)
|
||||
|
||||
def stats_domain_opened(self, domain):
|
||||
stats.set_value('start_time', datetime.datetime.utcnow(), domain=domain)
|
||||
stats.set_value('envinfo/host', stats.get_value('envinfo/host'), domain=domain)
|
||||
stats.inc_value('domain_count/opened')
|
||||
def stats_spider_opened(self, spider):
|
||||
stats.set_value('start_time', datetime.datetime.utcnow(), spider=spider)
|
||||
stats.set_value('envinfo/host', stats.get_value('envinfo/host'), spider=spider)
|
||||
stats.inc_value('spider_count/opened')
|
||||
|
||||
def stats_domain_closing(self, domain, reason):
|
||||
stats.set_value('finish_time', datetime.datetime.utcnow(), domain=domain)
|
||||
stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, domain=domain)
|
||||
stats.inc_value('domain_count/%s' % reason, domain=domain)
|
||||
def stats_spider_closing(self, spider, reason):
|
||||
stats.set_value('finish_time', datetime.datetime.utcnow(), spider=spider)
|
||||
stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, spider=spider)
|
||||
stats.inc_value('spider_count/%s' % reason, spider=spider)
|
||||
|
||||
def item_scraped(self, item, spider):
|
||||
stats.inc_value('item_scraped_count', domain=spider.domain_name)
|
||||
stats.inc_value('item_scraped_count', spider=spider)
|
||||
stats.inc_value('item_scraped_count')
|
||||
|
||||
def item_passed(self, item, spider):
|
||||
stats.inc_value('item_passed_count', domain=spider.domain_name)
|
||||
stats.inc_value('item_passed_count', spider=spider)
|
||||
stats.inc_value('item_passed_count')
|
||||
|
||||
def item_dropped(self, item, spider, exception):
|
||||
reason = exception.__class__.__name__
|
||||
stats.inc_value('item_dropped_count', domain=spider.domain_name)
|
||||
stats.inc_value('item_dropped_reasons_count/%s' % reason, domain=spider.domain_name)
|
||||
stats.inc_value('item_dropped_count', spider=spider)
|
||||
stats.inc_value('item_dropped_reasons_count/%s' % reason, spider=spider)
|
||||
stats.inc_value('item_dropped_count')
|
||||
|
|
|
|||
|
|
@ -5,37 +5,30 @@ from scrapy.stats import stats
|
|||
from scrapy.conf import settings
|
||||
|
||||
class DownloaderStats(object):
|
||||
"""DownloaderStats store stats of all requests, responses and
|
||||
exceptions that pass through it.
|
||||
|
||||
To use this middleware you must enable the DOWNLOADER_STATS setting.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
if not settings.getbool('DOWNLOADER_STATS'):
|
||||
raise NotConfigured
|
||||
|
||||
def process_request(self, request, spider):
|
||||
domain = spider.domain_name
|
||||
stats.inc_value('downloader/request_count')
|
||||
stats.inc_value('downloader/request_count', domain=domain)
|
||||
stats.inc_value('downloader/request_method_count/%s' % request.method, domain=domain)
|
||||
stats.inc_value('downloader/request_count', spider=spider)
|
||||
stats.inc_value('downloader/request_method_count/%s' % request.method, spider=spider)
|
||||
reqlen = len(request_httprepr(request))
|
||||
stats.inc_value('downloader/request_bytes', reqlen, domain=domain)
|
||||
stats.inc_value('downloader/request_bytes', reqlen, spider=spider)
|
||||
stats.inc_value('downloader/request_bytes', reqlen)
|
||||
|
||||
def process_response(self, request, response, spider):
|
||||
domain = spider.domain_name
|
||||
stats.inc_value('downloader/response_count')
|
||||
stats.inc_value('downloader/response_count', domain=domain)
|
||||
stats.inc_value('downloader/response_status_count/%s' % response.status, domain=domain)
|
||||
stats.inc_value('downloader/response_count', spider=spider)
|
||||
stats.inc_value('downloader/response_status_count/%s' % response.status, spider=spider)
|
||||
reslen = len(response_httprepr(response))
|
||||
stats.inc_value('downloader/response_bytes', reslen, domain=domain)
|
||||
stats.inc_value('downloader/response_bytes', reslen, spider=spider)
|
||||
stats.inc_value('downloader/response_bytes', reslen)
|
||||
return response
|
||||
|
||||
def process_exception(self, request, exception, spider):
|
||||
ex_class = "%s.%s" % (exception.__class__.__module__, exception.__class__.__name__)
|
||||
stats.inc_value('downloader/exception_count')
|
||||
stats.inc_value('downloader/exception_count', domain=spider.domain_name)
|
||||
stats.inc_value('downloader/exception_type_count/%s' % ex_class, domain=spider.domain_name)
|
||||
stats.inc_value('downloader/exception_count', spider=spider)
|
||||
stats.inc_value('downloader/exception_type_count/%s' % ex_class, spider=spider)
|
||||
|
|
|
|||
|
|
@ -53,11 +53,11 @@ class ItemSamplerPipeline(object):
|
|||
dispatcher.connect(self.engine_stopped, signal=signals.engine_stopped)
|
||||
|
||||
def process_item(self, spider, item):
|
||||
sampled = stats.get_value("items_sampled", 0, domain=spider.domain_name)
|
||||
sampled = stats.get_value("items_sampled", 0, spider=spider)
|
||||
if sampled < items_per_spider:
|
||||
self.items[item.guid] = item
|
||||
sampled += 1
|
||||
stats.set_value("items_sampled", sampled, domain=spider.domain_name)
|
||||
stats.set_value("items_sampled", sampled, spider=spider)
|
||||
log.msg("Sampled %s" % item, spider=spider, level=log.INFO)
|
||||
if close_spider and sampled == items_per_spider:
|
||||
scrapyengine.close_spider(spider)
|
||||
|
|
@ -71,7 +71,7 @@ class ItemSamplerPipeline(object):
|
|||
level=log.WARNING)
|
||||
|
||||
def spider_closed(self, spider, reason):
|
||||
if reason == 'finished' and not stats.get_value("items_sampled", domain=spider.domain_name):
|
||||
if reason == 'finished' and not stats.get_value("items_sampled", spider=spider):
|
||||
self.empty_domains.add(spider.domain_name)
|
||||
self.spiders_count += 1
|
||||
log.msg("Sampled %d domains so far (%d empty)" % (self.spiders_count, \
|
||||
|
|
@ -87,7 +87,7 @@ class ItemSamplerMiddleware(object):
|
|||
raise NotConfigured
|
||||
|
||||
def process_spider_input(self, response, spider):
|
||||
if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider:
|
||||
if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
|
||||
return []
|
||||
elif max_response_size and max_response_size > len(response_httprepr(response)):
|
||||
return []
|
||||
|
|
@ -100,7 +100,7 @@ class ItemSamplerMiddleware(object):
|
|||
else:
|
||||
items.append(r)
|
||||
|
||||
if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider:
|
||||
if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
|
||||
return []
|
||||
else:
|
||||
# TODO: this needs some revision, as keeping only the first item
|
||||
|
|
|
|||
|
|
@ -219,7 +219,7 @@ class ImagesPipeline(MediaPipeline):
|
|||
msg = 'Image (%s): Downloaded image from %s referred in <%s>' % \
|
||||
(status, request, referer)
|
||||
log.msg(msg, level=log.DEBUG, spider=info.spider)
|
||||
self.inc_stats(info.spider.domain_name, status)
|
||||
self.inc_stats(info.spider, status)
|
||||
|
||||
try:
|
||||
key = self.image_key(request.url)
|
||||
|
|
@ -258,7 +258,7 @@ class ImagesPipeline(MediaPipeline):
|
|||
referer = request.headers.get('Referer')
|
||||
log.msg('Image (uptodate): Downloaded %s from <%s> referred in <%s>' % \
|
||||
(self.MEDIA_NAME, request.url, referer), level=log.DEBUG, spider=info.spider)
|
||||
self.inc_stats(info.spider.domain_name, 'uptodate')
|
||||
self.inc_stats(info.spider, 'uptodate')
|
||||
|
||||
checksum = result.get('checksum', None)
|
||||
return {'url': request.url, 'path': key, 'checksum': checksum}
|
||||
|
|
@ -295,9 +295,9 @@ class ImagesPipeline(MediaPipeline):
|
|||
thumb_image, thumb_buf = self.convert_image(image, size)
|
||||
yield thumb_key, thumb_image, thumb_buf
|
||||
|
||||
def inc_stats(self, domain, status):
|
||||
stats.inc_value('image_count', domain=domain)
|
||||
stats.inc_value('image_status_count/%s' % status, domain=domain)
|
||||
def inc_stats(self, spider, status):
|
||||
stats.inc_value('image_count', spider=spider)
|
||||
stats.inc_value('image_status_count/%s' % status, spider=spider)
|
||||
|
||||
def convert_image(self, image, size=None):
|
||||
if image.mode != 'RGB':
|
||||
|
|
|
|||
|
|
@ -18,7 +18,6 @@ class DepthMiddleware(object):
|
|||
stats.set_value('envinfo/request_depth_limit', self.maxdepth)
|
||||
|
||||
def process_spider_output(self, response, result, spider):
|
||||
domain = spider.domain_name
|
||||
def _filter(request):
|
||||
if isinstance(request, Request):
|
||||
depth = response.request.meta['depth'] + 1
|
||||
|
|
@ -28,14 +27,14 @@ class DepthMiddleware(object):
|
|||
level=log.DEBUG, spider=spider)
|
||||
return False
|
||||
elif self.stats:
|
||||
stats.inc_value('request_depth_count/%s' % depth, domain=domain)
|
||||
if depth > stats.get_value('request_depth_max', 0, domain=domain):
|
||||
stats.set_value('request_depth_max', depth, domain=domain)
|
||||
stats.inc_value('request_depth_count/%s' % depth, spider=spider)
|
||||
if depth > stats.get_value('request_depth_max', 0, spider=spider):
|
||||
stats.set_value('request_depth_max', depth, spider=spider)
|
||||
return True
|
||||
|
||||
# base case (depth=0)
|
||||
if self.stats and 'depth' not in response.request.meta:
|
||||
response.request.meta['depth'] = 0
|
||||
stats.inc_value('request_depth_count/0', domain=domain)
|
||||
stats.inc_value('request_depth_count/0', spider=spider)
|
||||
|
||||
return (r for r in result or () if _filter(r))
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
"""
|
||||
StatsMailer extension sends an email when a domain finishes scraping.
|
||||
StatsMailer extension sends an email when a spider finishes scraping.
|
||||
|
||||
Use STATSMAILER_RCPTS setting to enable and give the recipient mail address
|
||||
"""
|
||||
|
|
@ -17,12 +17,12 @@ class StatsMailer(object):
|
|||
self.recipients = settings.getlist("STATSMAILER_RCPTS")
|
||||
if not self.recipients:
|
||||
raise NotConfigured
|
||||
dispatcher.connect(self.stats_domain_closed, signal=signals.stats_domain_closed)
|
||||
dispatcher.connect(self.stats_spider_closed, signal=signals.stats_spider_closed)
|
||||
|
||||
def stats_domain_closed(self, domain, domain_stats):
|
||||
def stats_spider_closed(self, spider, spider_stats):
|
||||
mail = MailSender()
|
||||
body = "Global stats\n\n"
|
||||
body += "\n".join("%-50s : %s" % i for i in stats.get_stats().items())
|
||||
body += "\n\n%s stats\n\n" % domain
|
||||
body += "\n".join("%-50s : %s" % i for i in domain_stats.items())
|
||||
mail.send(self.recipients, "Scrapy stats for: %s" % domain, body)
|
||||
body += "\n\n%s stats\n\n" % spider.domain_name
|
||||
body += "\n".join("%-50s : %s" % i for i in spider_stats.items())
|
||||
mail.send(self.recipients, "Scrapy stats for: %s" % spider.domain_name, body)
|
||||
|
|
|
|||
|
|
@ -22,9 +22,9 @@ class StatsDump(object):
|
|||
s = banner(self)
|
||||
s += "<h3>Global stats</h3>\n"
|
||||
s += stats_html_table(stats.get_stats())
|
||||
for domain in stats.list_domains():
|
||||
s += "<h3>%s</h3>\n" % domain
|
||||
s += stats_html_table(stats.get_stats(domain))
|
||||
for spider, spider_stats in stats.iter_spider_stats():
|
||||
s += "<h3>%s</h3>\n" % spider.domain_name
|
||||
s += stats_html_table(spider_stats)
|
||||
s += "</body>\n"
|
||||
s += "</html>\n"
|
||||
|
||||
|
|
|
|||
|
|
@ -45,20 +45,18 @@ class SpiderProfiler(object):
|
|||
r = function(*args, **kwargs)
|
||||
mafter = self._memusage()
|
||||
ct = time() - tbefore
|
||||
domain = spider.domain_name
|
||||
tcc = stats.get_value('profiling/total_callback_time', 0, domain=domain)
|
||||
sct = stats.get_value('profiling/slowest_callback_time', 0, domain=domain)
|
||||
stats.set_value('profiling/total_callback_time' % spider.domain_name, \
|
||||
tcc+ct, domain=domain)
|
||||
tcc = stats.get_value('profiling/total_callback_time', 0, spider=spider)
|
||||
sct = stats.get_value('profiling/slowest_callback_time', 0, spider=spider)
|
||||
stats.set_value('profiling/total_callback_time', tcc+ct, spider=spider)
|
||||
if ct > sct:
|
||||
stats.set_value('profiling/slowest_callback_time', ct, domain=domain)
|
||||
stats.set_value('profiling/slowest_callback_time', ct, spider=spider)
|
||||
stats.set_value('profiling/slowest_callback_name', function.__name__, \
|
||||
domain=domain)
|
||||
spider=spider)
|
||||
stats.set_value('profiling/slowest_callback_url', args[0].url, \
|
||||
domain=domain)
|
||||
spider=spider)
|
||||
if self._memusage:
|
||||
stats.inc_value('profiling/total_mem_allocated_in_callbacks', \
|
||||
count=mafter-mbefore, domain=domain)
|
||||
count=mafter-mbefore, spider=spider)
|
||||
return r
|
||||
return new_callback
|
||||
|
||||
|
|
|
|||
|
|
@ -244,7 +244,7 @@ class ExecutionEngine(object):
|
|||
|
||||
self.downloader.open_spider(spider)
|
||||
self.scraper.open_spider(spider)
|
||||
stats.open_domain(spider.domain_name)
|
||||
stats.open_spider(spider)
|
||||
|
||||
send_catch_log(signals.spider_opened, sender=self.__class__, spider=spider)
|
||||
|
||||
|
|
@ -305,7 +305,7 @@ class ExecutionEngine(object):
|
|||
reason = self.closing.pop(spider, 'finished')
|
||||
send_catch_log(signal=signals.spider_closed, sender=self.__class__, \
|
||||
spider=spider, reason=reason)
|
||||
stats.close_domain(spider.domain_name, reason=reason)
|
||||
stats.close_spider(spider, reason=reason)
|
||||
dfd = defer.maybeDeferred(spiders.close_spider, spider)
|
||||
dfd.addBoth(log.msg, "Spider closed (%s)" % reason, spider=spider)
|
||||
reactor.callLater(0, self._mainloop)
|
||||
|
|
|
|||
|
|
@ -87,17 +87,17 @@ class Scraper(object):
|
|||
def enqueue_scrape(self, response, request, spider):
|
||||
site = self.sites[spider]
|
||||
dfd = site.add_response_request(response, request)
|
||||
# FIXME: this can't be called here because the stats domain may be
|
||||
# FIXME: this can't be called here because the stats spider may be
|
||||
# already closed
|
||||
#stats.max_value('scraper/max_active_size', site.active_size, \
|
||||
# domain=spider.domain_name)
|
||||
# spider=spider)
|
||||
def finish_scraping(_):
|
||||
site.finish_response(response)
|
||||
self._scrape_next(spider, site)
|
||||
return _
|
||||
dfd.addBoth(finish_scraping)
|
||||
dfd.addErrback(log.err, 'Scraper bug processing %s' % request, \
|
||||
domain=spider.domain_name)
|
||||
spider=spider)
|
||||
self._scrape_next(spider, site)
|
||||
return dfd
|
||||
|
||||
|
|
@ -138,7 +138,7 @@ class Scraper(object):
|
|||
(request.url, referer, _failure)
|
||||
log.msg(msg, log.ERROR, spider=spider)
|
||||
stats.inc_value("spider_exceptions/%s" % _failure.value.__class__.__name__, \
|
||||
domain=spider.domain_name)
|
||||
spider=spider)
|
||||
|
||||
def handle_spider_output(self, result, request, response, spider):
|
||||
if not result:
|
||||
|
|
@ -152,7 +152,6 @@ class Scraper(object):
|
|||
from the given spider
|
||||
"""
|
||||
# TODO: keep closing state internally instead of checking engine
|
||||
domain = spider.domain_name
|
||||
if spider in self.engine.closing:
|
||||
return
|
||||
elif isinstance(output, Request):
|
||||
|
|
@ -165,10 +164,10 @@ class Scraper(object):
|
|||
send_catch_log(signal=signals.item_scraped, sender=self.__class__, \
|
||||
item=output, spider=spider, response=response)
|
||||
self.sites[spider].itemproc_size += 1
|
||||
# FIXME: this can't be called here because the stats domain may be
|
||||
# FIXME: this can't be called here because the stats spider may be
|
||||
# already closed
|
||||
#stats.max_value('scraper/max_itemproc_size', \
|
||||
# self.sites[domain].itemproc_size, domain=domain)
|
||||
# self.sites[spider].itemproc_size, spider=spider)
|
||||
dfd = self.itemproc.process_item(output, spider)
|
||||
dfd.addBoth(self._itemproc_finished, output, spider)
|
||||
return dfd
|
||||
|
|
@ -195,7 +194,6 @@ class Scraper(object):
|
|||
def _itemproc_finished(self, output, item, spider):
|
||||
"""ItemProcessor finished for the given ``item`` and returned ``output``
|
||||
"""
|
||||
domain = spider.domain_name
|
||||
self.sites[spider].itemproc_size -= 1
|
||||
if isinstance(output, Failure):
|
||||
ex = output.value
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@ import pprint
|
|||
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
|
||||
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \
|
||||
stats_domain_closed
|
||||
from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
|
||||
stats_spider_closed
|
||||
from scrapy.utils.signal import send_catch_log
|
||||
from scrapy.core import signals
|
||||
from scrapy import log
|
||||
|
|
@ -19,57 +19,57 @@ class StatsCollector(object):
|
|||
self._stats = {None: {}} # None is for global stats
|
||||
dispatcher.connect(self._engine_stopped, signal=signals.engine_stopped)
|
||||
|
||||
def get_value(self, key, default=None, domain=None):
|
||||
return self._stats[domain].get(key, default)
|
||||
def get_value(self, key, default=None, spider=None):
|
||||
return self._stats[spider].get(key, default)
|
||||
|
||||
def get_stats(self, domain=None):
|
||||
return self._stats[domain]
|
||||
def get_stats(self, spider=None):
|
||||
return self._stats[spider]
|
||||
|
||||
def set_value(self, key, value, domain=None):
|
||||
self._stats[domain][key] = value
|
||||
def set_value(self, key, value, spider=None):
|
||||
self._stats[spider][key] = value
|
||||
|
||||
def set_stats(self, stats, domain=None):
|
||||
self._stats[domain] = stats
|
||||
def set_stats(self, stats, spider=None):
|
||||
self._stats[spider] = stats
|
||||
|
||||
def inc_value(self, key, count=1, start=0, domain=None):
|
||||
d = self._stats[domain]
|
||||
def inc_value(self, key, count=1, start=0, spider=None):
|
||||
d = self._stats[spider]
|
||||
d[key] = d.setdefault(key, start) + count
|
||||
|
||||
def max_value(self, key, value, domain=None):
|
||||
d = self._stats[domain]
|
||||
def max_value(self, key, value, spider=None):
|
||||
d = self._stats[spider]
|
||||
d[key] = max(d.setdefault(key, value), value)
|
||||
|
||||
def min_value(self, key, value, domain=None):
|
||||
d = self._stats[domain]
|
||||
def min_value(self, key, value, spider=None):
|
||||
d = self._stats[spider]
|
||||
d[key] = min(d.setdefault(key, value), value)
|
||||
|
||||
def clear_stats(self, domain=None):
|
||||
self._stats[domain].clear()
|
||||
def clear_stats(self, spider=None):
|
||||
self._stats[spider].clear()
|
||||
|
||||
def list_domains(self):
|
||||
return [d for d in self._stats.keys() if d is not None]
|
||||
def iter_spider_stats(self):
|
||||
return [x for x in self._stats.iteritems() if x[0]]
|
||||
|
||||
def open_domain(self, domain):
|
||||
self._stats[domain] = {}
|
||||
send_catch_log(stats_domain_opened, domain=domain)
|
||||
def open_spider(self, spider):
|
||||
self._stats[spider] = {}
|
||||
send_catch_log(stats_spider_opened, spider=spider)
|
||||
|
||||
def close_domain(self, domain, reason):
|
||||
send_catch_log(stats_domain_closing, domain=domain, reason=reason)
|
||||
stats = self._stats.pop(domain)
|
||||
send_catch_log(stats_domain_closed, domain=domain, reason=reason, \
|
||||
domain_stats=stats)
|
||||
def close_spider(self, spider, reason):
|
||||
send_catch_log(stats_spider_closing, spider=spider, reason=reason)
|
||||
stats = self._stats.pop(spider)
|
||||
send_catch_log(stats_spider_closed, spider=spider, reason=reason, \
|
||||
spider_stats=stats)
|
||||
if self._dump:
|
||||
log.msg("Dumping domain stats:\n" + pprint.pformat(stats), \
|
||||
domain=domain)
|
||||
self._persist_stats(stats, domain)
|
||||
log.msg("Dumping spider stats:\n" + pprint.pformat(stats), \
|
||||
spider=spider)
|
||||
self._persist_stats(stats, spider)
|
||||
|
||||
def _engine_stopped(self):
|
||||
stats = self.get_stats()
|
||||
if self._dump:
|
||||
log.msg("Dumping global stats:\n" + pprint.pformat(stats))
|
||||
self._persist_stats(stats, domain=None)
|
||||
self._persist_stats(stats, spider=None)
|
||||
|
||||
def _persist_stats(self, stats, domain=None):
|
||||
def _persist_stats(self, stats, spider=None):
|
||||
pass
|
||||
|
||||
class MemoryStatsCollector(StatsCollector):
|
||||
|
|
@ -77,29 +77,30 @@ class MemoryStatsCollector(StatsCollector):
|
|||
def __init__(self):
|
||||
super(MemoryStatsCollector, self).__init__()
|
||||
self.domain_stats = {}
|
||||
|
||||
def _persist_stats(self, stats, domain=None):
|
||||
self.domain_stats[domain] = stats
|
||||
|
||||
def _persist_stats(self, stats, spider=None):
|
||||
if spider is not None:
|
||||
self.domain_stats[spider.domain_name] = stats
|
||||
|
||||
|
||||
class DummyStatsCollector(StatsCollector):
|
||||
|
||||
def get_value(self, key, default=None, domain=None):
|
||||
def get_value(self, key, default=None, spider=None):
|
||||
return default
|
||||
|
||||
def set_value(self, key, value, domain=None):
|
||||
def set_value(self, key, value, spider=None):
|
||||
pass
|
||||
|
||||
def set_stats(self, stats, domain=None):
|
||||
def set_stats(self, stats, spider=None):
|
||||
pass
|
||||
|
||||
def inc_value(self, key, count=1, start=0, domain=None):
|
||||
def inc_value(self, key, count=1, start=0, spider=None):
|
||||
pass
|
||||
|
||||
def max_value(self, key, value, domain=None):
|
||||
def max_value(self, key, value, spider=None):
|
||||
pass
|
||||
|
||||
def min_value(self, key, value, domain=None):
|
||||
def min_value(self, key, value, spider=None):
|
||||
pass
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -16,8 +16,8 @@ class MysqlStatsCollector(StatsCollector):
|
|||
mysqluri = settings['STATS_MYSQL_URI']
|
||||
self._mysql_conn = mysql_connect(mysqluri, use_unicode=False) if mysqluri else None
|
||||
|
||||
def _persist_stats(self, stats, domain=None):
|
||||
if domain is None: # only store domain-specific stats
|
||||
def _persist_stats(self, stats, spider=None):
|
||||
if spider is None: # only store spider-specific stats
|
||||
return
|
||||
if self._mysql_conn is None:
|
||||
return
|
||||
|
|
@ -27,5 +27,5 @@ class MysqlStatsCollector(StatsCollector):
|
|||
|
||||
c = self._mysql_conn.cursor()
|
||||
c.execute("INSERT INTO %s (domain,stored,data) VALUES (%%s,%%s,%%s)" % table, \
|
||||
(domain, stored, datas))
|
||||
(spider.domain_name, stored, datas))
|
||||
self._mysql_conn.commit()
|
||||
|
|
|
|||
|
|
@ -22,27 +22,27 @@ class SimpledbStatsCollector(StatsCollector):
|
|||
self._async = settings.getbool('STATS_SDB_ASYNC')
|
||||
connect_sdb().create_domain(self._sdbdomain)
|
||||
|
||||
def _persist_stats(self, stats, domain=None):
|
||||
if domain is None: # only store domain-specific stats
|
||||
def _persist_stats(self, stats, spider=None):
|
||||
if spider is None: # only store spider-specific stats
|
||||
return
|
||||
if not self._sdbdomain:
|
||||
return
|
||||
if self._async:
|
||||
dfd = threads.deferToThread(self._persist_to_sdb, domain, stats.copy())
|
||||
dfd = threads.deferToThread(self._persist_to_sdb, spider, stats.copy())
|
||||
dfd.addErrback(log.err, 'Error uploading stats to SimpleDB', \
|
||||
domain=domain)
|
||||
spider=spider)
|
||||
else:
|
||||
self._persist_to_sdb(domain, stats)
|
||||
self._persist_to_sdb(spider, stats)
|
||||
|
||||
def _persist_to_sdb(self, domain, stats):
|
||||
ts = self._get_timestamp(domain).isoformat()
|
||||
sdb_item_id = "%s_%s" % (domain, ts)
|
||||
def _persist_to_sdb(self, spider, stats):
|
||||
ts = self._get_timestamp(spider).isoformat()
|
||||
sdb_item_id = "%s_%s" % (spider.domain_name, ts)
|
||||
sdb_item = dict((k, self._to_sdb_value(v, k)) for k, v in stats.iteritems())
|
||||
sdb_item['domain'] = domain
|
||||
sdb_item['domain'] = spider.domain_name
|
||||
sdb_item['timestamp'] = self._to_sdb_value(ts)
|
||||
connect_sdb().put_attributes(self._sdbdomain, sdb_item_id, sdb_item)
|
||||
|
||||
def _get_timestamp(self, domain):
|
||||
def _get_timestamp(self, spider):
|
||||
return datetime.utcnow()
|
||||
|
||||
def _to_sdb_value(self, obj, key=None):
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
stats_domain_opened = object()
|
||||
stats_domain_closing = object()
|
||||
stats_domain_closed = object()
|
||||
stats_spider_opened = object()
|
||||
stats_spider_closing = object()
|
||||
stats_spider_closed = object()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
from unittest import TestCase
|
||||
|
||||
from scrapy.conf import settings
|
||||
from scrapy.contrib.downloadermiddleware.stats import DownloaderStats
|
||||
from scrapy.http import Request, Response
|
||||
from scrapy.spider import BaseSpider
|
||||
|
|
@ -10,11 +9,10 @@ from scrapy.stats import stats
|
|||
class TestDownloaderStats(TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.spider = BaseSpider()
|
||||
self.spider.domain_name = 'scrapytest.org'
|
||||
self.spider = BaseSpider('scrapytest.org')
|
||||
self.mw = DownloaderStats()
|
||||
|
||||
stats.open_domain(self.spider.domain_name)
|
||||
stats.open_spider(self.spider)
|
||||
|
||||
self.req = Request('scrapytest.org')
|
||||
self.res = Response('scrapytest.org', status=400)
|
||||
|
|
@ -22,18 +20,18 @@ class TestDownloaderStats(TestCase):
|
|||
def test_process_request(self):
|
||||
self.mw.process_request(self.req, self.spider)
|
||||
self.assertEqual(stats.get_value('downloader/request_count', \
|
||||
domain=self.spider.domain_name), 1)
|
||||
spider=self.spider), 1)
|
||||
|
||||
def test_process_response(self):
|
||||
self.mw.process_response(self.req, self.res, self.spider)
|
||||
self.assertEqual(stats.get_value('downloader/response_count', \
|
||||
domain=self.spider.domain_name), 1)
|
||||
spider=self.spider), 1)
|
||||
|
||||
def test_process_exception(self):
|
||||
self.mw.process_exception(self.req, Exception(), self.spider)
|
||||
self.assertEqual(stats.get_value('downloader/exception_count', \
|
||||
domain=self.spider.domain_name), 1)
|
||||
spider=self.spider), 1)
|
||||
|
||||
def tearDown(self):
|
||||
stats.close_domain(self.spider.domain_name, '')
|
||||
stats.close_spider(self.spider, '')
|
||||
|
||||
|
|
|
|||
|
|
@ -14,10 +14,9 @@ class TestDepthMiddleware(TestCase):
|
|||
settings.overrides['DEPTH_LIMIT'] = 1
|
||||
settings.overrides['DEPTH_STATS'] = True
|
||||
|
||||
self.spider = BaseSpider()
|
||||
self.spider.domain_name = 'scrapytest.org'
|
||||
self.spider = BaseSpider('scrapytest.org')
|
||||
|
||||
stats.open_domain(self.spider.domain_name)
|
||||
stats.open_spider(self.spider)
|
||||
|
||||
self.mw = DepthMiddleware()
|
||||
self.assertEquals(stats.get_value('envinfo/request_depth_limit'), 1)
|
||||
|
|
@ -31,8 +30,7 @@ class TestDepthMiddleware(TestCase):
|
|||
out = list(self.mw.process_spider_output(resp, result, self.spider))
|
||||
self.assertEquals(out, result)
|
||||
|
||||
rdc = stats.get_value('request_depth_count/1',
|
||||
domain=self.spider.domain_name)
|
||||
rdc = stats.get_value('request_depth_count/1', spider=self.spider)
|
||||
self.assertEquals(rdc, 1)
|
||||
|
||||
req.meta['depth'] = 1
|
||||
|
|
@ -40,8 +38,7 @@ class TestDepthMiddleware(TestCase):
|
|||
out2 = list(self.mw.process_spider_output(resp, result, self.spider))
|
||||
self.assertEquals(out2, [])
|
||||
|
||||
rdm = stats.get_value('request_depth_max',
|
||||
domain=self.spider.domain_name)
|
||||
rdm = stats.get_value('request_depth_max', spider=self.spider)
|
||||
self.assertEquals(rdm, 1)
|
||||
|
||||
def tearDown(self):
|
||||
|
|
@ -49,5 +46,5 @@ class TestDepthMiddleware(TestCase):
|
|||
del settings.overrides['DEPTH_STATS']
|
||||
settings.disabled = True
|
||||
|
||||
stats.close_domain(self.spider.domain_name, '')
|
||||
stats.close_spider(self.spider, '')
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,16 @@
|
|||
import unittest
|
||||
|
||||
from scrapy.spider import BaseSpider
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
from scrapy.stats.collector import StatsCollector, DummyStatsCollector
|
||||
from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \
|
||||
stats_domain_closed
|
||||
from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
|
||||
stats_spider_closed
|
||||
|
||||
class StatsCollectorTest(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.spider = BaseSpider()
|
||||
|
||||
def test_collector(self):
|
||||
stats = StatsCollector()
|
||||
self.assertEqual(stats.get_stats(), {})
|
||||
|
|
@ -43,44 +47,45 @@ class StatsCollectorTest(unittest.TestCase):
|
|||
stats.inc_value('v1')
|
||||
stats.max_value('v2', 100)
|
||||
stats.min_value('v3', 100)
|
||||
stats.open_domain('a')
|
||||
stats.set_value('test', 'value', domain='a')
|
||||
stats.open_spider('a')
|
||||
stats.set_value('test', 'value', spider=self.spider)
|
||||
self.assertEqual(stats.get_stats(), {})
|
||||
self.assertEqual(stats.get_stats('a'), {})
|
||||
|
||||
def test_signals(self):
|
||||
signals_catched = set()
|
||||
|
||||
def domain_opened(domain):
|
||||
assert domain == 'example.com'
|
||||
signals_catched.add(stats_domain_opened)
|
||||
def spider_opened(spider):
|
||||
assert spider is self.spider
|
||||
signals_catched.add(stats_spider_opened)
|
||||
|
||||
def domain_closing(domain, reason):
|
||||
assert domain == 'example.com'
|
||||
def spider_closing(spider, reason):
|
||||
assert spider is self.spider
|
||||
assert reason == 'testing'
|
||||
signals_catched.add(stats_domain_closing)
|
||||
signals_catched.add(stats_spider_closing)
|
||||
|
||||
def domain_closed(domain, reason, domain_stats):
|
||||
assert domain == 'example.com'
|
||||
def spider_closed(spider, reason, spider_stats):
|
||||
assert spider is self.spider
|
||||
assert reason == 'testing'
|
||||
assert domain_stats == {'test': 1}
|
||||
signals_catched.add(stats_domain_closed)
|
||||
assert spider_stats == {'test': 1}
|
||||
signals_catched.add(stats_spider_closed)
|
||||
|
||||
dispatcher.connect(domain_opened, signal=stats_domain_opened)
|
||||
dispatcher.connect(domain_closing, signal=stats_domain_closing)
|
||||
dispatcher.connect(domain_closed, signal=stats_domain_closed)
|
||||
dispatcher.connect(spider_opened, signal=stats_spider_opened)
|
||||
dispatcher.connect(spider_closing, signal=stats_spider_closing)
|
||||
dispatcher.connect(spider_closed, signal=stats_spider_closed)
|
||||
|
||||
stats = StatsCollector()
|
||||
stats.open_domain('example.com')
|
||||
stats.set_value('test', 1, domain='example.com')
|
||||
stats.close_domain('example.com', 'testing')
|
||||
assert stats_domain_opened in signals_catched
|
||||
assert stats_domain_closing in signals_catched
|
||||
assert stats_domain_closed in signals_catched
|
||||
stats.open_spider(self.spider)
|
||||
stats.set_value('test', 1, spider=self.spider)
|
||||
self.assertEqual([(self.spider, {'test': 1})], list(stats.iter_spider_stats()))
|
||||
stats.close_spider(self.spider, 'testing')
|
||||
assert stats_spider_opened in signals_catched
|
||||
assert stats_spider_closing in signals_catched
|
||||
assert stats_spider_closed in signals_catched
|
||||
|
||||
dispatcher.disconnect(domain_opened, signal=stats_domain_opened)
|
||||
dispatcher.disconnect(domain_closing, signal=stats_domain_closing)
|
||||
dispatcher.disconnect(domain_closed, signal=stats_domain_closed)
|
||||
dispatcher.disconnect(spider_opened, signal=stats_spider_opened)
|
||||
dispatcher.disconnect(spider_closing, signal=stats_spider_closing)
|
||||
dispatcher.disconnect(spider_closed, signal=stats_spider_closed)
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
Loading…
Reference in New Issue