* removed starters from engine
* moved schedulermiddleware to scheduler
* raise RuntimeError when trying to open/close a scheduler domain twice
* removed dupesfilter singleton and spidermw dupefilter middleware
--HG--
extra : rebase_source : e4c3ad4b970cbc8f532bc751ba9d8a944ca16be5
spider level. A new spider (InitSpider, not yet documented) was added to
provide initialization facilities.
Also renamed make_request_from_url to make_requests_from_url and allowed it to
return iterables.
Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
Execution Manager:
* added control_reactor argument to delegate external twisted
reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods
Execution engine:
* added control_reactor argument to to delegate external twisted
reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
for legibilty
Scheduler:
* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
constructor (domain prioritizers will be refactored later anyway) and
simplified Scrapy Manager code thanks to this.
Added make_request_from_url method to BaseSpider, splitting funtionality to
create requests from URLs which was previously done all in start_requests.
in this changeset:
* remove HttpException from engine and core exceptions
* replace dwmw ErrorPages with spidermw HttpError
* bugfix image pipeline media_to_download method when stat_key returns None
* documented
* moved from scrapy.contrib.downloadermiddleware.cache.CacheMiddleware to
scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware
* settings prefix changed from CACHE2_ to HTTPCACHE_
--HG--
rename : scrapy/contrib/downloadermiddleware/cache.py => scrapy/contrib/downloadermiddleware/httpcache.py
Pablo Hoffman
Daniel Grana
daniel
Daniel Grana