RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

SimpledbStatsCollector: moved domain creation to constructor

This commit is contained in:
Pablo Hoffman 2009-07-17 12:49:53 -03:00
parent 97a854a53a
commit 92b746d866
2 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/topics/stats.rst
View File

@ -201,7 +201,8 @@ SimpledbStatsCollector
A Stats collector which persists stats to `Amazon SimpleDB`_, using one
SimpleDB item per scraping run (ie. it keeps history of all scraping runs).
The data is persisted to the SimpleDB domain specified by the
:setting:`STATS_SDB_DOMAIN` setting.
:setting:`STATS_SDB_DOMAIN` setting. The domain will be created if it
doesn't exist.
In addition to the existing stats keys the following keys are added at
persitance time:

3
scrapy/stats/collector/simpledb.py
View File

@ -20,6 +20,8 @@ class SimpledbStatsCollector(StatsCollector):
super(SimpledbStatsCollector, self).__init__()
self._sdbdomain = settings['STATS_SDB_DOMAIN']
self._async = settings.getbool('STATS_SDB_ASYNC')
sdb = boto.connect_sdb()
sdb.create_domain(self._sdbdomain)
def close_domain(self, domain, reason):
if self._sdbdomain:
@ -39,7 +41,6 @@ class SimpledbStatsCollector(StatsCollector):
sdb_item['domain'] = domain
sdb_item['timestamp'] = self._to_sdb_value(ts)
sdb = boto.connect_sdb()
sdb.create_domain(self._sdbdomain)
sdb.batch_put_attributes(self._sdbdomain, {sdb_item_id: sdb_item})
def _to_sdb_value(self, obj):