RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/scrapy
History
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 		2011-08-02 11:57:55 -03:00
..
commands More core changes: 2011-07-15 15:18:39 -03:00
contrib Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
contrib_exp Fixed DjangoItem to work properly with auto-generated 2011-06-12 01:41:09 -03:00
core Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
http Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
selector avoid stripping trailing spaces on lxml-based selectors. closes #270 2010-10-27 21:39:28 -02:00
settings Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
templates removed unused import from 'crawl' spider template 2011-07-20 01:04:16 -03:00
tests Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
utils Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
xlib Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled. 2011-07-27 03:45:15 -03:00
__init__.py Bumped version to 0.13 2011-01-02 17:21:31 -02:00
cmdline.py A couple of changes to fix #303: 2011-01-05 15:59:43 -02:00
command.py Added support for setting exit code in Scrapy commands. Closes #248 2010-09-23 13:43:21 -03:00
conf.py Removed hacky SCRAPY_SETTINGS_DISABLED environment variable 2010-09-22 16:08:18 -03:00
crawler.py Added cached DNS resolver based on old caching resolver extension from scrapy.contrib.resolver. This new one is *not* an extension, it comes builtin and always enabled. 2011-07-27 03:45:15 -03:00
dupefilter.py Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
exceptions.py added CloseSpider exception, to manually close spiders 2011-07-12 14:24:10 -03:00
extension.py Made ExtensionManager a subclass of MiddlewareManager 2010-08-22 05:33:08 -03:00
interfaces.py moved spider queues to scrapyd 2011-07-19 19:39:27 -03:00
item.py Changed default representation of items to pretty-printed dicts. This improves 2011-06-03 01:13:01 -03:00
link.py added fragment attribute ot Link object 2011-07-27 12:28:36 -03:00
linkextractor.py Some Link extractor improvements: 2011-05-18 12:32:34 -03:00
log.py Don't silence warnings after log is started. Closes #246 2010-09-22 01:00:34 -03:00
logformatter.py improved download errors propagation to the spiders, and removed no longer needed code to simplify 2011-07-13 14:10:05 -03:00
mail.py removed (somewhat hacky) MAIL_DEBUG setting 2010-08-22 22:42:00 -03:00
middleware.py A couple of changes to fix #303: 2011-01-05 15:59:43 -02:00
project.py Refactoring of Crawler, Commands, Execution Queue and Spider Manager: 2010-08-25 19:24:36 -03:00
resolver.py minor comments adjustments 2011-07-27 03:48:38 -03:00
shell.py More core changes: 2011-07-15 15:18:39 -03:00
signals.py Merged item passed and item scraped concepts, as they have often proved 2011-06-03 01:13:00 -03:00
spider.py Added support for unifying access to per-spider settings. Refs #245 2010-09-22 16:09:13 -03:00
spidermanager.py spidermanager: more detailed error message now that scrapy crawl command will raise the exception directly 2011-07-26 19:05:29 -03:00
squeue.py Initial support for a persistent scheduler, to support pausing and resuming 2011-08-02 11:57:55 -03:00
stats.py Splitted stats collector classes from stats collection facility (#204) 2010-08-22 01:24:07 -03:00
statscol.py call stats collector engine_stopped() after the engine is closed (to make sure all data from extensions has been collected), and added that method to documented api 2011-06-06 03:12:40 -03:00
telnet.py Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-11-10 13:36:05 -02:00
webservice.py Fixed bug in addversion.json with old Twisted versions. Closes #293 2010-11-25 12:12:42 -02:00