Log messages count are grouped per LEVEL like:
'log_count/DEBUG': 8,
'log_count/ERROR': 1,
'log_count/INFO': 4,
This required passing the Crawler object to scrapy logging facility, so
the scrapy.log module underwent a few changes which should be backwards
compatible (or as backwards compatible as they could be).
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
Commands changes:
* removed (somewhat hacky) --init argument from settings command
* added set_crawler method to Commands, and a ``crawler`` property that returns
a configured crawler. This way, commands that don't require a crawler (such
as startproject) won't need to configure one.
Execution Queue changes:
* changed SERVICE_QUEUE_FILE setting to SQLITE_DB
* removed SERVICE_QUEUE setting
* added QUEUE_CLASS setting for defining the class to use for the execution queue
* added SERVER_QUEUE_CLASS setting for defining the class to use for the
execution queue in server mode (runserver command)
Spider Manager changes:
* simplified SpiderManager API by removing the load() method
* added from_settings classmethod to SpiderManager
* added spider_modules constructor argument to SpiderManager
Crawler changes:
* added install() method to Crawler (to install it in scrapy.project) and
uninstall() to remove it
* use CrawlerProcess.install() in scrapy.cmdline
* use crawler.install() and crawler.uninstall() in tests that a crawler in
scrapy.project
* make telnet console and webservice play nicer with twisted by stopping
listening when then engine goes down
* refactored Scrapy engine tests - it no longer uses the crawler singleton.
Closes#215.
Julia Medina
Rolando Espinoza La fuente
Alex Cepoi
Pablo Hoffman