Althought this backwards compatibility is more complex, it avoid modules
from failing when importing scrapy.conf, if they are not run through
"scrapy" command (such as when running tests on scrapy projects code).
scrapy.http.response.text, scrapy.selector
- fixed bug with scrapy.conf.settings backwards compatibility support
- added facility to notify (and provide some guidelines) about deprecated/obsolete settings
* improved detection of inside-project environments
* make list command faster (by only instantiating the spider manger)
* print a warning when extensions (middlewares, etc) are disabled with a message on NotConfigured exception
* assert that scrapy configuration hasn't been loaded in scrapyd.runner
* simplified IgnoreRequest exception, to avoid loading settings when importing scrapy.exceptions
* added test to make sure certain modules don't cause scrapy.conf module to be
loaded, to ensure the scrapyd runner bootstraping performs properly
Commands changes:
* removed (somewhat hacky) --init argument from settings command
* added set_crawler method to Commands, and a ``crawler`` property that returns
a configured crawler. This way, commands that don't require a crawler (such
as startproject) won't need to configure one.
Execution Queue changes:
* changed SERVICE_QUEUE_FILE setting to SQLITE_DB
* removed SERVICE_QUEUE setting
* added QUEUE_CLASS setting for defining the class to use for the execution queue
* added SERVER_QUEUE_CLASS setting for defining the class to use for the
execution queue in server mode (runserver command)
Spider Manager changes:
* simplified SpiderManager API by removing the load() method
* added from_settings classmethod to SpiderManager
* added spider_modules constructor argument to SpiderManager
Crawler changes:
* added install() method to Crawler (to install it in scrapy.project) and
uninstall() to remove it
* use CrawlerProcess.install() in scrapy.cmdline
* use crawler.install() and crawler.uninstall() in tests that a crawler in
scrapy.project
* make telnet console and webservice play nicer with twisted by stopping
listening when then engine goes down
* refactored Scrapy engine tests - it no longer uses the crawler singleton.
Closes#215.
into this class and leaving the Crawler class free of any reactor control.
This allows embedding the Scrapy crawler in other Twisted applications, or any
Twisted asynchronous code.
The "scrapy" command will use the new CrawlerProcess class, which resembles the
behaviour of the old (all-in-one) Crawler class.
Closes#214.
Also moved log.start() call from Crawler class to scrapy.cmdline module.
Detail of changes:
* Moved scrapy.core.manager.ExecutionManager class to scrapy.crawler.Crawler
* Added scrapy.project.crawler singleton to reference a singleton instance of
Crawler class (previously known as scrapymanager)
* Left an alias scrapy.core.manager.scrapymanager to scrapy.project.crawler for
backwards compatibility (to be removed in Scrapy 0.11)
Mikhail Korobov
nyov
Julia Medina
Felix Yan
alexanderlukanin13
Alex Cepoi
Jordi Llonch
Pablo Hoffman