Log messages count are grouped per LEVEL like:
'log_count/DEBUG': 8,
'log_count/ERROR': 1,
'log_count/INFO': 4,
This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
* added Twisted log observer for Scrapy, with unittests
* use numeric values from Python logging module for log levels
* removed scrapy.log.exc() function - use scrapy.log.err() instead
* removed logmessage_received signal - write a (twisted) log observer instead
* dropped support for obsolete `domain` argument
* dropped support for old setting names: LOGLEVEL, LOGFILE (replaced by LOG_LEVEL, LOG_FILE)
* deprecated `component` argument
* Rename BaseSpider.domain_name to BaseSpider.name
This patch implements the domain_name to name change in BaseSpider class and
change all spider instantiations to use the new attribute.
* Add allowed_domains to spider
This patch implements the merging of spider.domain_name and
spider.extra_domain_names in spider.allowed_domains for offsite checking
purposes.
Note that spider.domain_name is not touched by this patch, only not used.
* Remove spider.domain_name references from scrapy.stats
* Rename domain_stats to spider_stats in MemoryStatsCollector
* Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
* Rename domain_stats_history table to spider_data_history and rename domain
field to spider in MysqlStatsCollector
* Refactor genspider command
The new signature for genspider is: genspider [options] <domain_name>.
Genspider uses domain_name for spider name and for the module name.
* Remove spider.domain_name references
* Update crawl command signature <spider|url>
* docs: updated references to domain_name
* examples/experimental: use spider.name
* genspider: require <name> <domain>
* spidermanager: renamed crawl_domain to crawl_spider_name
* spiderctl: updated references of *domain* to spider
* added backward compatiblity with legacy spider's attributes
'domain_name' and 'extra_domain_names'
- added graceful shutdown (with one ^C) and forced shutdown (two ^C)
- added optional loglevel to IgnoreRequest exception
- calling Spider Manager close_domain() function from engine to avoid race
conditions caused by signal dispatch order
- moved twisted reactor controlling logic outside the engine and into the
scrapy manager
- made log.err function respect the current log level filter
Alexander Chekunkov
Julia Medina
Mikhail Korobov
Paul Tremberth
Nikolay Golub
Alex Cepoi
Pablo Hoffman
Daniel Graña
Rolando Espinoza La fuente