Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming ... crawls. * requests are serialized (using marshal by default) and stored on disk, using one queue per priority * request priorities must be integers now * breadh-first and depth-first crawling orders can now be configured through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with SCHEDULER_ORDER was kept. * requests that can't be serialized (for example, non serializable callbacks) are always kept in memory queues * adapted crawl spider to work with persitent scheduler		2011-08-02 11:57:55 -03:00
..
downloader	restored support for download delays after downlaoder refactoring, also restored support for spider attributes: max_concurrent_requests and download_delay	2011-07-27 15:14:27 -03:00
__init__.py	mv scrapy/trunk to root as part of svn2hg migration	2009-05-06 15:55:17 -03:00
engine.py	imported patch scheduler_single_spider.patch	2011-07-31 03:32:25 -03:00
scheduler.py	Initial support for a persistent scheduler, to support pausing and resuming	2011-08-02 11:57:55 -03:00
scraper.py	fixed bug with scraper KeyError's on some ConnectionLost errors. closes #334	2011-07-25 12:24:26 -03:00
spidermw.py	Fixed spider middleware order for methods: process_spider_output, process_spider_exception	2010-09-07 12:59:02 -03:00