diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst index 3b3026654..971a33364 100644 --- a/docs/topics/stats.rst +++ b/docs/topics/stats.rst @@ -8,9 +8,9 @@ Overview ======== Scrapy provides a convenient service for collecting stats in the form of -key/values, both globally and per spider/domain. It's called the Stats -Collector, and it's a singleton which can be imported and used quickly, as -illustrated by the examples in the :ref:`topics-stats-usecases` section below. +key/values, both globally and per spider. It's called the Stats Collector, and +it's a singleton which can be imported and used quickly, as illustrated by the +examples in the :ref:`topics-stats-usecases` section below. The stats collection is enabled by default but can be disabled through the :setting:`STATS_ENABLED` setting. @@ -26,10 +26,10 @@ using the Stats Collector from. Another feature of the Stats Collector is that it's very efficient (when enabled) and extremely efficient (almost unnoticeable) when disabled. -The Stats Collector keeps one stats table per open spider/domain and one global -stats table. You can't set or get stats from a closed domain, but the -domain-specific stats table is automatically opened when the spider is opened, -and closed when the spider is closed. +The Stats Collector keeps one stats table per open spider and one global stats +table. You can't set or get stats from a closed spider, but the spider-specific +stats table is automatically opened when the spider is opened, and closed when +the spider is closed. .. _topics-stats-usecases: @@ -61,36 +61,38 @@ Get global stat value:: >>> stats.get_value('spiders_crawled') 8 -Get all global stats from a given domain:: +Get all global stats (ie. not particular to any spider):: >>> stats.get_stats() {'hostname': 'localhost', 'spiders_crawled': 8} -Set domain/spider specific stat value (domains must be opened first, but this +Set spider specific stat value (spider stats must be opened first, but this task is handled automatically by the Scrapy engine):: - stats.set_value('start_time', datetime.now(), domain='example.com') + stats.set_value('start_time', datetime.now(), spider=some_spider) -Increment domain-specific stat value:: +Where ``some_spider`` is a :class:`~scrapy.spider.BaseSpider` object. - stats.inc_value('pages_crawled', domain='example.com') +Increment spider-specific stat value:: -Set domain-specific stat value only if greater than previous:: + stats.inc_value('pages_crawled', spider=some_spider) - stats.max_value('max_items_scraped', value, domain='example.com') +Set spider-specific stat value only if greater than previous:: -Set domain-specific stat value only if lower than previous:: + stats.max_value('max_items_scraped', value, spider=some_spider) - stats.min_value('min_free_memory_percent', value, domain='example.com') +Set spider-specific stat value only if lower than previous:: -Get domain-specific stat value:: + stats.min_value('min_free_memory_percent', value, spider=some_spider) - >>> stats.get_value('pages_crawled', domain='example.com') +Get spider-specific stat value:: + + >>> stats.get_value('pages_crawled', spider=some_spider) 1238 -Get all stats from a given domain:: +Get all stats from a given spider:: - >>> stats.get_stats('pages_crawled', domain='example.com') + >>> stats.get_stats('pages_crawled', spider=some_spider) {'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)} .. _topics-stats-ref: @@ -108,74 +110,79 @@ class (which they all inherit from). .. class:: StatsCollector - .. method:: get_value(key, default=None, domain=None) + .. method:: get_value(key, default=None, spider=None) Return the value for the given stats key or default if it doesn't exist. - If domain is ``None`` the global stats table is consulted, other the - domain specific one is. If the domain is not yet opened a ``KeyError`` + If spider is ``None`` the global stats table is consulted, otherwise the + spider specific one is. If the spider is not yet opened a ``KeyError`` exception is raised. - .. method:: get_stats(domain=None) + .. method:: get_stats(spider=None) - Get all stats from the given domain/spider (if domain is given) or all - global stats otherwise, as a dict. If domain is not opened ``KeyError`` - is raied. + Get all stats from the given spider (if spider is given) or all global + stats otherwise, as a dict. If spider is not opened ``KeyError`` is + raied. - .. method:: set_value(key, value, domain=None) + .. method:: set_value(key, value, spider=None) Set the given value for the given stats key on the global stats (if - domain is not given) or the domain-specific stats (if domain is given), + spider is not given) or the spider-specific stats (if spider is given), which must be opened or a ``KeyError`` will be raised. - .. method:: set_stats(stats, domain=None) + .. method:: set_stats(stats, spider=None) - Set the given stats (as a dict) for the given domain. If the domain is + Set the given stats (as a dict) for the given spider. If the spider is not opened a ``KeyError`` will be raised. - .. method:: inc_value(key, count=1, start=0, domain=None) + .. method:: inc_value(key, count=1, start=0, spider=None) Increment the value of the given stats key, by the given count, - assuming the start value given (when it's not set). If domain is not - given the global stats table is used, otherwise the domain-specific + assuming the start value given (when it's not set). If spider is not + given the global stats table is used, otherwise the spider-specific stats table is used, which must be opened or a ``KeyError`` will be raised. - .. method:: max_value(key, value, domain=None) + .. method:: max_value(key, value, spider=None) Set the given value for the given key only if current value for the same key is lower than value. If there is no current value for the - given key, the value is always set. If domain is not given the global - stats table is used, otherwise the domain-specific stats table is used, + given key, the value is always set. If spider is not given the global + stats table is used, otherwise the spider-specific stats table is used, which must be opened or a KeyError will be raised. - .. method:: min_value(key, value, domain=None) + .. method:: min_value(key, value, spider=None) Set the given value for the given key only if current value for the same key is greater than value. If there is no current value for the - given key, the value is always set. If domain is not given the global - stats table is used, otherwise the domain-specific stats table is used, + given key, the value is always set. If spider is not given the global + stats table is used, otherwise the spider-specific stats table is used, which must be opened or a KeyError will be raised. - .. method:: clear_stats(domain=None) + .. method:: clear_stats(spider=None) - Clear all global stats (if domain is not given) or all domain-specific - stats if domain is given, in which case it must be opened or a + Clear all global stats (if spider is not given) or all spider-specific + stats if spider is given, in which case it must be opened or a ``KeyError`` will be raised. - .. method:: list_domains() + .. method:: iter_spider_stats() - Return a list of all opened domains. + Return a iterator over ``(spider, spider_stats)`` for each open spider + currently tracked by the stats collector, where ``spider_stats`` is the + dict containing all spider-specific stats. - .. method:: open_domain(domain) + Global stats are not included in the iterator. If you want to get + those, use :meth:`get_stats` method. - Open the given domain for stats collection. This method must be called - prior to working with any stats specific to that domain, but this task + .. method:: open_spider(spider) + + Open the given spider for stats collection. This method must be called + prior to working with any stats specific to that spider, but this task is handled automatically by the Scrapy engine. - .. method:: close_domain(domain) + .. method:: close_spider(spider) - Close the given domain. After this is called, no more specific stats - for this domain can be accessed. This method is called automatically on + Close the given spider. After this is called, no more specific stats + for this spider can be accessed. This method is called automatically on the :signal:`spider_closed` signal. Available Stats Collectors @@ -196,15 +203,16 @@ MemoryStatsCollector .. class:: MemoryStatsCollector A simple stats collector that keeps the stats of the last scraping run (for - each domain) in memory, which can be accessed through the ``domain_stats`` - attribute + each spider) in memory, after they're closed. The stats can be accessed + through the :attr:`domain_stats` attribute, which is a dict keyed by spider + domain name. This is the default Stats Collector used in Scrapy. .. attribute:: domain_stats - A dict of dicts (keyed by domain) containing the stats of the last - scraping run for each domain. + A dict of dicts (keyed by spider domain name) containing the stats of + the last scraping run for each domain. DummyStatsCollector ------------------- @@ -283,41 +291,41 @@ functionality: .. module:: scrapy.stats.signals :synopsis: Stats Collector signals -.. signal:: stats_domain_opened -.. function:: stats_domain_opened(domain) +.. signal:: stats_spider_opened +.. function:: stats_spider_opened(spider) - Sent right after the stats domain is opened. You can use this signal to add - startup stats for domain (example: start time). + Sent right after the stats spider is opened. You can use this signal to add + startup stats for spider (example: start time). - :param domain: the stats domain just opened - :type domain: str + :param spider: the stats spider just opened + :type spider: str -.. signal:: stats_domain_closing -.. function:: stats_domain_closing(domain, reason) +.. signal:: stats_spider_closing +.. function:: stats_spider_closing(spider, reason) - Sent just before the stats domain is closed. You can use this signal to add + Sent just before the stats spider is closed. You can use this signal to add some closing stats (example: finish time). - :param domain: the stats domain about to be closed - :type domain: str + :param spider: the stats spider about to be closed + :type spider: str - :param reason: the reason why the domain is being closed. See + :param reason: the reason why the spider is being closed. See :signal:`spider_closed` signal for more info. :type reason: str -.. signal:: stats_domain_closed -.. function:: stats_domain_closed(domain, reason, domain_stats) +.. signal:: stats_spider_closed +.. function:: stats_spider_closed(spider, reason, spider_stats) - Sent right after the stats domain is closed. You can use this signal to - collect resources, but not to add any more stats as the stats domain has - already been close (use :signal:`stats_domain_closing` for that instead). + Sent right after the stats spider is closed. You can use this signal to + collect resources, but not to add any more stats as the stats spider has + already been close (use :signal:`stats_spider_closing` for that instead). - :param domain: the stats domain just closed - :type domain: str + :param spider: the stats spider just closed + :type spider: str - :param reason: the reason why the domain was closed. See + :param reason: the reason why the spider was closed. See :signal:`spider_closed` signal for more info. :type reason: str - :param domain_stats: the stats of the domain just closed. + :param spider_stats: the stats of the spider just closed. :type reason: dict diff --git a/scrapy/contrib/corestats.py b/scrapy/contrib/corestats.py index 08819abbb..79e9c73e5 100644 --- a/scrapy/contrib/corestats.py +++ b/scrapy/contrib/corestats.py @@ -10,7 +10,7 @@ from scrapy.xlib.pydispatch import dispatcher from scrapy.core import signals from scrapy.stats import stats -from scrapy.stats.signals import stats_domain_opened, stats_domain_closing +from scrapy.stats.signals import stats_spider_opened, stats_spider_closing from scrapy.conf import settings class CoreStats(object): @@ -22,32 +22,32 @@ class CoreStats(object): stats.set_value('envinfo/logfile', settings['LOG_FILE']) stats.set_value('envinfo/pid', os.getpid()) - dispatcher.connect(self.stats_domain_opened, signal=stats_domain_opened) - dispatcher.connect(self.stats_domain_closing, signal=stats_domain_closing) + dispatcher.connect(self.stats_spider_opened, signal=stats_spider_opened) + dispatcher.connect(self.stats_spider_closing, signal=stats_spider_closing) dispatcher.connect(self.item_scraped, signal=signals.item_scraped) dispatcher.connect(self.item_passed, signal=signals.item_passed) dispatcher.connect(self.item_dropped, signal=signals.item_dropped) - def stats_domain_opened(self, domain): - stats.set_value('start_time', datetime.datetime.utcnow(), domain=domain) - stats.set_value('envinfo/host', stats.get_value('envinfo/host'), domain=domain) - stats.inc_value('domain_count/opened') + def stats_spider_opened(self, spider): + stats.set_value('start_time', datetime.datetime.utcnow(), spider=spider) + stats.set_value('envinfo/host', stats.get_value('envinfo/host'), spider=spider) + stats.inc_value('spider_count/opened') - def stats_domain_closing(self, domain, reason): - stats.set_value('finish_time', datetime.datetime.utcnow(), domain=domain) - stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, domain=domain) - stats.inc_value('domain_count/%s' % reason, domain=domain) + def stats_spider_closing(self, spider, reason): + stats.set_value('finish_time', datetime.datetime.utcnow(), spider=spider) + stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, spider=spider) + stats.inc_value('spider_count/%s' % reason, spider=spider) def item_scraped(self, item, spider): - stats.inc_value('item_scraped_count', domain=spider.domain_name) + stats.inc_value('item_scraped_count', spider=spider) stats.inc_value('item_scraped_count') def item_passed(self, item, spider): - stats.inc_value('item_passed_count', domain=spider.domain_name) + stats.inc_value('item_passed_count', spider=spider) stats.inc_value('item_passed_count') def item_dropped(self, item, spider, exception): reason = exception.__class__.__name__ - stats.inc_value('item_dropped_count', domain=spider.domain_name) - stats.inc_value('item_dropped_reasons_count/%s' % reason, domain=spider.domain_name) + stats.inc_value('item_dropped_count', spider=spider) + stats.inc_value('item_dropped_reasons_count/%s' % reason, spider=spider) stats.inc_value('item_dropped_count') diff --git a/scrapy/contrib/downloadermiddleware/stats.py b/scrapy/contrib/downloadermiddleware/stats.py index 0a4631696..64d9e92fc 100644 --- a/scrapy/contrib/downloadermiddleware/stats.py +++ b/scrapy/contrib/downloadermiddleware/stats.py @@ -5,37 +5,30 @@ from scrapy.stats import stats from scrapy.conf import settings class DownloaderStats(object): - """DownloaderStats store stats of all requests, responses and - exceptions that pass through it. - - To use this middleware you must enable the DOWNLOADER_STATS setting. - """ def __init__(self): if not settings.getbool('DOWNLOADER_STATS'): raise NotConfigured def process_request(self, request, spider): - domain = spider.domain_name stats.inc_value('downloader/request_count') - stats.inc_value('downloader/request_count', domain=domain) - stats.inc_value('downloader/request_method_count/%s' % request.method, domain=domain) + stats.inc_value('downloader/request_count', spider=spider) + stats.inc_value('downloader/request_method_count/%s' % request.method, spider=spider) reqlen = len(request_httprepr(request)) - stats.inc_value('downloader/request_bytes', reqlen, domain=domain) + stats.inc_value('downloader/request_bytes', reqlen, spider=spider) stats.inc_value('downloader/request_bytes', reqlen) def process_response(self, request, response, spider): - domain = spider.domain_name stats.inc_value('downloader/response_count') - stats.inc_value('downloader/response_count', domain=domain) - stats.inc_value('downloader/response_status_count/%s' % response.status, domain=domain) + stats.inc_value('downloader/response_count', spider=spider) + stats.inc_value('downloader/response_status_count/%s' % response.status, spider=spider) reslen = len(response_httprepr(response)) - stats.inc_value('downloader/response_bytes', reslen, domain=domain) + stats.inc_value('downloader/response_bytes', reslen, spider=spider) stats.inc_value('downloader/response_bytes', reslen) return response def process_exception(self, request, exception, spider): ex_class = "%s.%s" % (exception.__class__.__module__, exception.__class__.__name__) stats.inc_value('downloader/exception_count') - stats.inc_value('downloader/exception_count', domain=spider.domain_name) - stats.inc_value('downloader/exception_type_count/%s' % ex_class, domain=spider.domain_name) + stats.inc_value('downloader/exception_count', spider=spider) + stats.inc_value('downloader/exception_type_count/%s' % ex_class, spider=spider) diff --git a/scrapy/contrib/itemsampler.py b/scrapy/contrib/itemsampler.py index 68bbd0680..6021bbf5a 100644 --- a/scrapy/contrib/itemsampler.py +++ b/scrapy/contrib/itemsampler.py @@ -53,11 +53,11 @@ class ItemSamplerPipeline(object): dispatcher.connect(self.engine_stopped, signal=signals.engine_stopped) def process_item(self, spider, item): - sampled = stats.get_value("items_sampled", 0, domain=spider.domain_name) + sampled = stats.get_value("items_sampled", 0, spider=spider) if sampled < items_per_spider: self.items[item.guid] = item sampled += 1 - stats.set_value("items_sampled", sampled, domain=spider.domain_name) + stats.set_value("items_sampled", sampled, spider=spider) log.msg("Sampled %s" % item, spider=spider, level=log.INFO) if close_spider and sampled == items_per_spider: scrapyengine.close_spider(spider) @@ -71,7 +71,7 @@ class ItemSamplerPipeline(object): level=log.WARNING) def spider_closed(self, spider, reason): - if reason == 'finished' and not stats.get_value("items_sampled", domain=spider.domain_name): + if reason == 'finished' and not stats.get_value("items_sampled", spider=spider): self.empty_domains.add(spider.domain_name) self.spiders_count += 1 log.msg("Sampled %d domains so far (%d empty)" % (self.spiders_count, \ @@ -87,7 +87,7 @@ class ItemSamplerMiddleware(object): raise NotConfigured def process_spider_input(self, response, spider): - if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider: + if stats.get_value("items_sampled", spider=spider) >= items_per_spider: return [] elif max_response_size and max_response_size > len(response_httprepr(response)): return [] @@ -100,7 +100,7 @@ class ItemSamplerMiddleware(object): else: items.append(r) - if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider: + if stats.get_value("items_sampled", spider=spider) >= items_per_spider: return [] else: # TODO: this needs some revision, as keeping only the first item diff --git a/scrapy/contrib/pipeline/images.py b/scrapy/contrib/pipeline/images.py index 4fc137c03..3a93553dd 100644 --- a/scrapy/contrib/pipeline/images.py +++ b/scrapy/contrib/pipeline/images.py @@ -219,7 +219,7 @@ class ImagesPipeline(MediaPipeline): msg = 'Image (%s): Downloaded image from %s referred in <%s>' % \ (status, request, referer) log.msg(msg, level=log.DEBUG, spider=info.spider) - self.inc_stats(info.spider.domain_name, status) + self.inc_stats(info.spider, status) try: key = self.image_key(request.url) @@ -258,7 +258,7 @@ class ImagesPipeline(MediaPipeline): referer = request.headers.get('Referer') log.msg('Image (uptodate): Downloaded %s from <%s> referred in <%s>' % \ (self.MEDIA_NAME, request.url, referer), level=log.DEBUG, spider=info.spider) - self.inc_stats(info.spider.domain_name, 'uptodate') + self.inc_stats(info.spider, 'uptodate') checksum = result.get('checksum', None) return {'url': request.url, 'path': key, 'checksum': checksum} @@ -295,9 +295,9 @@ class ImagesPipeline(MediaPipeline): thumb_image, thumb_buf = self.convert_image(image, size) yield thumb_key, thumb_image, thumb_buf - def inc_stats(self, domain, status): - stats.inc_value('image_count', domain=domain) - stats.inc_value('image_status_count/%s' % status, domain=domain) + def inc_stats(self, spider, status): + stats.inc_value('image_count', spider=spider) + stats.inc_value('image_status_count/%s' % status, spider=spider) def convert_image(self, image, size=None): if image.mode != 'RGB': diff --git a/scrapy/contrib/spidermiddleware/depth.py b/scrapy/contrib/spidermiddleware/depth.py index 9e89d2f39..4b15285d7 100644 --- a/scrapy/contrib/spidermiddleware/depth.py +++ b/scrapy/contrib/spidermiddleware/depth.py @@ -18,7 +18,6 @@ class DepthMiddleware(object): stats.set_value('envinfo/request_depth_limit', self.maxdepth) def process_spider_output(self, response, result, spider): - domain = spider.domain_name def _filter(request): if isinstance(request, Request): depth = response.request.meta['depth'] + 1 @@ -28,14 +27,14 @@ class DepthMiddleware(object): level=log.DEBUG, spider=spider) return False elif self.stats: - stats.inc_value('request_depth_count/%s' % depth, domain=domain) - if depth > stats.get_value('request_depth_max', 0, domain=domain): - stats.set_value('request_depth_max', depth, domain=domain) + stats.inc_value('request_depth_count/%s' % depth, spider=spider) + if depth > stats.get_value('request_depth_max', 0, spider=spider): + stats.set_value('request_depth_max', depth, spider=spider) return True # base case (depth=0) if self.stats and 'depth' not in response.request.meta: response.request.meta['depth'] = 0 - stats.inc_value('request_depth_count/0', domain=domain) + stats.inc_value('request_depth_count/0', spider=spider) return (r for r in result or () if _filter(r)) diff --git a/scrapy/contrib/statsmailer.py b/scrapy/contrib/statsmailer.py index 4a580bf5c..c2185e0a7 100644 --- a/scrapy/contrib/statsmailer.py +++ b/scrapy/contrib/statsmailer.py @@ -1,5 +1,5 @@ """ -StatsMailer extension sends an email when a domain finishes scraping. +StatsMailer extension sends an email when a spider finishes scraping. Use STATSMAILER_RCPTS setting to enable and give the recipient mail address """ @@ -17,12 +17,12 @@ class StatsMailer(object): self.recipients = settings.getlist("STATSMAILER_RCPTS") if not self.recipients: raise NotConfigured - dispatcher.connect(self.stats_domain_closed, signal=signals.stats_domain_closed) + dispatcher.connect(self.stats_spider_closed, signal=signals.stats_spider_closed) - def stats_domain_closed(self, domain, domain_stats): + def stats_spider_closed(self, spider, spider_stats): mail = MailSender() body = "Global stats\n\n" body += "\n".join("%-50s : %s" % i for i in stats.get_stats().items()) - body += "\n\n%s stats\n\n" % domain - body += "\n".join("%-50s : %s" % i for i in domain_stats.items()) - mail.send(self.recipients, "Scrapy stats for: %s" % domain, body) + body += "\n\n%s stats\n\n" % spider.domain_name + body += "\n".join("%-50s : %s" % i for i in spider_stats.items()) + mail.send(self.recipients, "Scrapy stats for: %s" % spider.domain_name, body) diff --git a/scrapy/contrib/webconsole/stats.py b/scrapy/contrib/webconsole/stats.py index 8f3ea9b82..837cbcf81 100644 --- a/scrapy/contrib/webconsole/stats.py +++ b/scrapy/contrib/webconsole/stats.py @@ -22,9 +22,9 @@ class StatsDump(object): s = banner(self) s += "

Global stats

\n" s += stats_html_table(stats.get_stats()) - for domain in stats.list_domains(): - s += "

%s

\n" % domain - s += stats_html_table(stats.get_stats(domain)) + for spider, spider_stats in stats.iter_spider_stats(): + s += "

%s

\n" % spider.domain_name + s += stats_html_table(spider_stats) s += "\n" s += "\n" diff --git a/scrapy/contrib_exp/spiderprofiler.py b/scrapy/contrib_exp/spiderprofiler.py index b7c3ace7c..0587c6997 100644 --- a/scrapy/contrib_exp/spiderprofiler.py +++ b/scrapy/contrib_exp/spiderprofiler.py @@ -45,20 +45,18 @@ class SpiderProfiler(object): r = function(*args, **kwargs) mafter = self._memusage() ct = time() - tbefore - domain = spider.domain_name - tcc = stats.get_value('profiling/total_callback_time', 0, domain=domain) - sct = stats.get_value('profiling/slowest_callback_time', 0, domain=domain) - stats.set_value('profiling/total_callback_time' % spider.domain_name, \ - tcc+ct, domain=domain) + tcc = stats.get_value('profiling/total_callback_time', 0, spider=spider) + sct = stats.get_value('profiling/slowest_callback_time', 0, spider=spider) + stats.set_value('profiling/total_callback_time', tcc+ct, spider=spider) if ct > sct: - stats.set_value('profiling/slowest_callback_time', ct, domain=domain) + stats.set_value('profiling/slowest_callback_time', ct, spider=spider) stats.set_value('profiling/slowest_callback_name', function.__name__, \ - domain=domain) + spider=spider) stats.set_value('profiling/slowest_callback_url', args[0].url, \ - domain=domain) + spider=spider) if self._memusage: stats.inc_value('profiling/total_mem_allocated_in_callbacks', \ - count=mafter-mbefore, domain=domain) + count=mafter-mbefore, spider=spider) return r return new_callback diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py index b54f5a2ae..29f8581c8 100644 --- a/scrapy/core/engine.py +++ b/scrapy/core/engine.py @@ -244,7 +244,7 @@ class ExecutionEngine(object): self.downloader.open_spider(spider) self.scraper.open_spider(spider) - stats.open_domain(spider.domain_name) + stats.open_spider(spider) send_catch_log(signals.spider_opened, sender=self.__class__, spider=spider) @@ -305,7 +305,7 @@ class ExecutionEngine(object): reason = self.closing.pop(spider, 'finished') send_catch_log(signal=signals.spider_closed, sender=self.__class__, \ spider=spider, reason=reason) - stats.close_domain(spider.domain_name, reason=reason) + stats.close_spider(spider, reason=reason) dfd = defer.maybeDeferred(spiders.close_spider, spider) dfd.addBoth(log.msg, "Spider closed (%s)" % reason, spider=spider) reactor.callLater(0, self._mainloop) diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py index 06d9487c7..8a0618810 100644 --- a/scrapy/core/scraper.py +++ b/scrapy/core/scraper.py @@ -87,17 +87,17 @@ class Scraper(object): def enqueue_scrape(self, response, request, spider): site = self.sites[spider] dfd = site.add_response_request(response, request) - # FIXME: this can't be called here because the stats domain may be + # FIXME: this can't be called here because the stats spider may be # already closed #stats.max_value('scraper/max_active_size', site.active_size, \ - # domain=spider.domain_name) + # spider=spider) def finish_scraping(_): site.finish_response(response) self._scrape_next(spider, site) return _ dfd.addBoth(finish_scraping) dfd.addErrback(log.err, 'Scraper bug processing %s' % request, \ - domain=spider.domain_name) + spider=spider) self._scrape_next(spider, site) return dfd @@ -138,7 +138,7 @@ class Scraper(object): (request.url, referer, _failure) log.msg(msg, log.ERROR, spider=spider) stats.inc_value("spider_exceptions/%s" % _failure.value.__class__.__name__, \ - domain=spider.domain_name) + spider=spider) def handle_spider_output(self, result, request, response, spider): if not result: @@ -152,7 +152,6 @@ class Scraper(object): from the given spider """ # TODO: keep closing state internally instead of checking engine - domain = spider.domain_name if spider in self.engine.closing: return elif isinstance(output, Request): @@ -165,10 +164,10 @@ class Scraper(object): send_catch_log(signal=signals.item_scraped, sender=self.__class__, \ item=output, spider=spider, response=response) self.sites[spider].itemproc_size += 1 - # FIXME: this can't be called here because the stats domain may be + # FIXME: this can't be called here because the stats spider may be # already closed #stats.max_value('scraper/max_itemproc_size', \ - # self.sites[domain].itemproc_size, domain=domain) + # self.sites[spider].itemproc_size, spider=spider) dfd = self.itemproc.process_item(output, spider) dfd.addBoth(self._itemproc_finished, output, spider) return dfd @@ -195,7 +194,6 @@ class Scraper(object): def _itemproc_finished(self, output, item, spider): """ItemProcessor finished for the given ``item`` and returned ``output`` """ - domain = spider.domain_name self.sites[spider].itemproc_size -= 1 if isinstance(output, Failure): ex = output.value diff --git a/scrapy/stats/collector/__init__.py b/scrapy/stats/collector/__init__.py index 4dc60a1d2..f41916818 100644 --- a/scrapy/stats/collector/__init__.py +++ b/scrapy/stats/collector/__init__.py @@ -5,8 +5,8 @@ import pprint from scrapy.xlib.pydispatch import dispatcher -from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \ - stats_domain_closed +from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \ + stats_spider_closed from scrapy.utils.signal import send_catch_log from scrapy.core import signals from scrapy import log @@ -19,57 +19,57 @@ class StatsCollector(object): self._stats = {None: {}} # None is for global stats dispatcher.connect(self._engine_stopped, signal=signals.engine_stopped) - def get_value(self, key, default=None, domain=None): - return self._stats[domain].get(key, default) + def get_value(self, key, default=None, spider=None): + return self._stats[spider].get(key, default) - def get_stats(self, domain=None): - return self._stats[domain] + def get_stats(self, spider=None): + return self._stats[spider] - def set_value(self, key, value, domain=None): - self._stats[domain][key] = value + def set_value(self, key, value, spider=None): + self._stats[spider][key] = value - def set_stats(self, stats, domain=None): - self._stats[domain] = stats + def set_stats(self, stats, spider=None): + self._stats[spider] = stats - def inc_value(self, key, count=1, start=0, domain=None): - d = self._stats[domain] + def inc_value(self, key, count=1, start=0, spider=None): + d = self._stats[spider] d[key] = d.setdefault(key, start) + count - def max_value(self, key, value, domain=None): - d = self._stats[domain] + def max_value(self, key, value, spider=None): + d = self._stats[spider] d[key] = max(d.setdefault(key, value), value) - def min_value(self, key, value, domain=None): - d = self._stats[domain] + def min_value(self, key, value, spider=None): + d = self._stats[spider] d[key] = min(d.setdefault(key, value), value) - def clear_stats(self, domain=None): - self._stats[domain].clear() + def clear_stats(self, spider=None): + self._stats[spider].clear() - def list_domains(self): - return [d for d in self._stats.keys() if d is not None] + def iter_spider_stats(self): + return [x for x in self._stats.iteritems() if x[0]] - def open_domain(self, domain): - self._stats[domain] = {} - send_catch_log(stats_domain_opened, domain=domain) + def open_spider(self, spider): + self._stats[spider] = {} + send_catch_log(stats_spider_opened, spider=spider) - def close_domain(self, domain, reason): - send_catch_log(stats_domain_closing, domain=domain, reason=reason) - stats = self._stats.pop(domain) - send_catch_log(stats_domain_closed, domain=domain, reason=reason, \ - domain_stats=stats) + def close_spider(self, spider, reason): + send_catch_log(stats_spider_closing, spider=spider, reason=reason) + stats = self._stats.pop(spider) + send_catch_log(stats_spider_closed, spider=spider, reason=reason, \ + spider_stats=stats) if self._dump: - log.msg("Dumping domain stats:\n" + pprint.pformat(stats), \ - domain=domain) - self._persist_stats(stats, domain) + log.msg("Dumping spider stats:\n" + pprint.pformat(stats), \ + spider=spider) + self._persist_stats(stats, spider) def _engine_stopped(self): stats = self.get_stats() if self._dump: log.msg("Dumping global stats:\n" + pprint.pformat(stats)) - self._persist_stats(stats, domain=None) + self._persist_stats(stats, spider=None) - def _persist_stats(self, stats, domain=None): + def _persist_stats(self, stats, spider=None): pass class MemoryStatsCollector(StatsCollector): @@ -77,29 +77,30 @@ class MemoryStatsCollector(StatsCollector): def __init__(self): super(MemoryStatsCollector, self).__init__() self.domain_stats = {} - - def _persist_stats(self, stats, domain=None): - self.domain_stats[domain] = stats + + def _persist_stats(self, stats, spider=None): + if spider is not None: + self.domain_stats[spider.domain_name] = stats class DummyStatsCollector(StatsCollector): - def get_value(self, key, default=None, domain=None): + def get_value(self, key, default=None, spider=None): return default - def set_value(self, key, value, domain=None): + def set_value(self, key, value, spider=None): pass - def set_stats(self, stats, domain=None): + def set_stats(self, stats, spider=None): pass - def inc_value(self, key, count=1, start=0, domain=None): + def inc_value(self, key, count=1, start=0, spider=None): pass - def max_value(self, key, value, domain=None): + def max_value(self, key, value, spider=None): pass - def min_value(self, key, value, domain=None): + def min_value(self, key, value, spider=None): pass diff --git a/scrapy/stats/collector/mysql.py b/scrapy/stats/collector/mysql.py index 3b20377aa..2cf145347 100644 --- a/scrapy/stats/collector/mysql.py +++ b/scrapy/stats/collector/mysql.py @@ -16,8 +16,8 @@ class MysqlStatsCollector(StatsCollector): mysqluri = settings['STATS_MYSQL_URI'] self._mysql_conn = mysql_connect(mysqluri, use_unicode=False) if mysqluri else None - def _persist_stats(self, stats, domain=None): - if domain is None: # only store domain-specific stats + def _persist_stats(self, stats, spider=None): + if spider is None: # only store spider-specific stats return if self._mysql_conn is None: return @@ -27,5 +27,5 @@ class MysqlStatsCollector(StatsCollector): c = self._mysql_conn.cursor() c.execute("INSERT INTO %s (domain,stored,data) VALUES (%%s,%%s,%%s)" % table, \ - (domain, stored, datas)) + (spider.domain_name, stored, datas)) self._mysql_conn.commit() diff --git a/scrapy/stats/collector/simpledb.py b/scrapy/stats/collector/simpledb.py index 030fe3caf..850558442 100644 --- a/scrapy/stats/collector/simpledb.py +++ b/scrapy/stats/collector/simpledb.py @@ -22,27 +22,27 @@ class SimpledbStatsCollector(StatsCollector): self._async = settings.getbool('STATS_SDB_ASYNC') connect_sdb().create_domain(self._sdbdomain) - def _persist_stats(self, stats, domain=None): - if domain is None: # only store domain-specific stats + def _persist_stats(self, stats, spider=None): + if spider is None: # only store spider-specific stats return if not self._sdbdomain: return if self._async: - dfd = threads.deferToThread(self._persist_to_sdb, domain, stats.copy()) + dfd = threads.deferToThread(self._persist_to_sdb, spider, stats.copy()) dfd.addErrback(log.err, 'Error uploading stats to SimpleDB', \ - domain=domain) + spider=spider) else: - self._persist_to_sdb(domain, stats) + self._persist_to_sdb(spider, stats) - def _persist_to_sdb(self, domain, stats): - ts = self._get_timestamp(domain).isoformat() - sdb_item_id = "%s_%s" % (domain, ts) + def _persist_to_sdb(self, spider, stats): + ts = self._get_timestamp(spider).isoformat() + sdb_item_id = "%s_%s" % (spider.domain_name, ts) sdb_item = dict((k, self._to_sdb_value(v, k)) for k, v in stats.iteritems()) - sdb_item['domain'] = domain + sdb_item['domain'] = spider.domain_name sdb_item['timestamp'] = self._to_sdb_value(ts) connect_sdb().put_attributes(self._sdbdomain, sdb_item_id, sdb_item) - def _get_timestamp(self, domain): + def _get_timestamp(self, spider): return datetime.utcnow() def _to_sdb_value(self, obj, key=None): diff --git a/scrapy/stats/signals.py b/scrapy/stats/signals.py index fb363c173..19d3eb295 100644 --- a/scrapy/stats/signals.py +++ b/scrapy/stats/signals.py @@ -1,3 +1,3 @@ -stats_domain_opened = object() -stats_domain_closing = object() -stats_domain_closed = object() +stats_spider_opened = object() +stats_spider_closing = object() +stats_spider_closed = object() diff --git a/scrapy/tests/test_downloadermiddleware_stats.py b/scrapy/tests/test_downloadermiddleware_stats.py index 076caee91..bd1fcfd91 100644 --- a/scrapy/tests/test_downloadermiddleware_stats.py +++ b/scrapy/tests/test_downloadermiddleware_stats.py @@ -1,6 +1,5 @@ from unittest import TestCase -from scrapy.conf import settings from scrapy.contrib.downloadermiddleware.stats import DownloaderStats from scrapy.http import Request, Response from scrapy.spider import BaseSpider @@ -10,11 +9,10 @@ from scrapy.stats import stats class TestDownloaderStats(TestCase): def setUp(self): - self.spider = BaseSpider() - self.spider.domain_name = 'scrapytest.org' + self.spider = BaseSpider('scrapytest.org') self.mw = DownloaderStats() - stats.open_domain(self.spider.domain_name) + stats.open_spider(self.spider) self.req = Request('scrapytest.org') self.res = Response('scrapytest.org', status=400) @@ -22,18 +20,18 @@ class TestDownloaderStats(TestCase): def test_process_request(self): self.mw.process_request(self.req, self.spider) self.assertEqual(stats.get_value('downloader/request_count', \ - domain=self.spider.domain_name), 1) + spider=self.spider), 1) def test_process_response(self): self.mw.process_response(self.req, self.res, self.spider) self.assertEqual(stats.get_value('downloader/response_count', \ - domain=self.spider.domain_name), 1) + spider=self.spider), 1) def test_process_exception(self): self.mw.process_exception(self.req, Exception(), self.spider) self.assertEqual(stats.get_value('downloader/exception_count', \ - domain=self.spider.domain_name), 1) + spider=self.spider), 1) def tearDown(self): - stats.close_domain(self.spider.domain_name, '') + stats.close_spider(self.spider, '') diff --git a/scrapy/tests/test_spidermiddleware_depth.py b/scrapy/tests/test_spidermiddleware_depth.py index 580e331e8..e52dffb94 100644 --- a/scrapy/tests/test_spidermiddleware_depth.py +++ b/scrapy/tests/test_spidermiddleware_depth.py @@ -14,10 +14,9 @@ class TestDepthMiddleware(TestCase): settings.overrides['DEPTH_LIMIT'] = 1 settings.overrides['DEPTH_STATS'] = True - self.spider = BaseSpider() - self.spider.domain_name = 'scrapytest.org' + self.spider = BaseSpider('scrapytest.org') - stats.open_domain(self.spider.domain_name) + stats.open_spider(self.spider) self.mw = DepthMiddleware() self.assertEquals(stats.get_value('envinfo/request_depth_limit'), 1) @@ -31,8 +30,7 @@ class TestDepthMiddleware(TestCase): out = list(self.mw.process_spider_output(resp, result, self.spider)) self.assertEquals(out, result) - rdc = stats.get_value('request_depth_count/1', - domain=self.spider.domain_name) + rdc = stats.get_value('request_depth_count/1', spider=self.spider) self.assertEquals(rdc, 1) req.meta['depth'] = 1 @@ -40,8 +38,7 @@ class TestDepthMiddleware(TestCase): out2 = list(self.mw.process_spider_output(resp, result, self.spider)) self.assertEquals(out2, []) - rdm = stats.get_value('request_depth_max', - domain=self.spider.domain_name) + rdm = stats.get_value('request_depth_max', spider=self.spider) self.assertEquals(rdm, 1) def tearDown(self): @@ -49,5 +46,5 @@ class TestDepthMiddleware(TestCase): del settings.overrides['DEPTH_STATS'] settings.disabled = True - stats.close_domain(self.spider.domain_name, '') + stats.close_spider(self.spider, '') diff --git a/scrapy/tests/test_stats.py b/scrapy/tests/test_stats.py index ed1712c6c..612ee0eaa 100644 --- a/scrapy/tests/test_stats.py +++ b/scrapy/tests/test_stats.py @@ -1,12 +1,16 @@ import unittest +from scrapy.spider import BaseSpider from scrapy.xlib.pydispatch import dispatcher from scrapy.stats.collector import StatsCollector, DummyStatsCollector -from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \ - stats_domain_closed +from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \ + stats_spider_closed class StatsCollectorTest(unittest.TestCase): + def setUp(self): + self.spider = BaseSpider() + def test_collector(self): stats = StatsCollector() self.assertEqual(stats.get_stats(), {}) @@ -43,44 +47,45 @@ class StatsCollectorTest(unittest.TestCase): stats.inc_value('v1') stats.max_value('v2', 100) stats.min_value('v3', 100) - stats.open_domain('a') - stats.set_value('test', 'value', domain='a') + stats.open_spider('a') + stats.set_value('test', 'value', spider=self.spider) self.assertEqual(stats.get_stats(), {}) self.assertEqual(stats.get_stats('a'), {}) def test_signals(self): signals_catched = set() - def domain_opened(domain): - assert domain == 'example.com' - signals_catched.add(stats_domain_opened) + def spider_opened(spider): + assert spider is self.spider + signals_catched.add(stats_spider_opened) - def domain_closing(domain, reason): - assert domain == 'example.com' + def spider_closing(spider, reason): + assert spider is self.spider assert reason == 'testing' - signals_catched.add(stats_domain_closing) + signals_catched.add(stats_spider_closing) - def domain_closed(domain, reason, domain_stats): - assert domain == 'example.com' + def spider_closed(spider, reason, spider_stats): + assert spider is self.spider assert reason == 'testing' - assert domain_stats == {'test': 1} - signals_catched.add(stats_domain_closed) + assert spider_stats == {'test': 1} + signals_catched.add(stats_spider_closed) - dispatcher.connect(domain_opened, signal=stats_domain_opened) - dispatcher.connect(domain_closing, signal=stats_domain_closing) - dispatcher.connect(domain_closed, signal=stats_domain_closed) + dispatcher.connect(spider_opened, signal=stats_spider_opened) + dispatcher.connect(spider_closing, signal=stats_spider_closing) + dispatcher.connect(spider_closed, signal=stats_spider_closed) stats = StatsCollector() - stats.open_domain('example.com') - stats.set_value('test', 1, domain='example.com') - stats.close_domain('example.com', 'testing') - assert stats_domain_opened in signals_catched - assert stats_domain_closing in signals_catched - assert stats_domain_closed in signals_catched + stats.open_spider(self.spider) + stats.set_value('test', 1, spider=self.spider) + self.assertEqual([(self.spider, {'test': 1})], list(stats.iter_spider_stats())) + stats.close_spider(self.spider, 'testing') + assert stats_spider_opened in signals_catched + assert stats_spider_closing in signals_catched + assert stats_spider_closed in signals_catched - dispatcher.disconnect(domain_opened, signal=stats_domain_opened) - dispatcher.disconnect(domain_closing, signal=stats_domain_closing) - dispatcher.disconnect(domain_closed, signal=stats_domain_closed) + dispatcher.disconnect(spider_opened, signal=stats_spider_opened) + dispatcher.disconnect(spider_closing, signal=stats_spider_closing) + dispatcher.disconnect(spider_closed, signal=stats_spider_closed) if __name__ == "__main__": unittest.main()