RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
2,217 Commits 42 Branches 132 Tags 168 MiB
Python 99.8%
HTML 0.1%
Go to file
Pablo Hoffman 8fc78c4d0a Refactoring of Crawler, Commands, Execution Queue and Spider Manager: 
Commands changes:

* removed (somewhat hacky) --init argument from settings command
* added set_crawler method to Commands, and a ``crawler`` property that returns
  a configured crawler. This way, commands that don't require a crawler (such
  as startproject) won't need to configure one.

Execution Queue changes:

* changed SERVICE_QUEUE_FILE setting to SQLITE_DB
* removed SERVICE_QUEUE setting
* added QUEUE_CLASS setting for defining the class to use for the execution queue
* added SERVER_QUEUE_CLASS setting for defining the class to use for the
  execution queue in server mode (runserver command)

Spider Manager changes:

* simplified SpiderManager API by removing the load() method
* added from_settings classmethod to SpiderManager
* added spider_modules constructor argument to SpiderManager

Crawler changes:

* added install() method to Crawler (to install it in scrapy.project) and
  uninstall() to remove it
* use CrawlerProcess.install() in scrapy.cmdline
* use crawler.install() and crawler.uninstall() in tests that a crawler in
  scrapy.project
* make telnet console and webservice play nicer with twisted by stopping
  listening when then engine goes down
* refactored Scrapy engine tests - it no longer uses the crawler singleton.
  Closes #215.
 		2010-08-25 19:24:36 -03:00
bin Removed scrapy-sqs script, as it has been superseded by the new scrapy 'queue' command 2010-08-23 22:04:49 -03:00
debian Removed scrapy-sqs script, as it has been superseded by the new scrapy 'queue' command 2010-08-23 22:04:49 -03:00
docs Added documentation for Ubuntu packages. Refs #211 2010-08-23 21:28:32 -03:00
examples example projects: added scrapy.cfg and removed scrapy-ctl.py 2010-08-21 04:56:19 -03:00
extras Minor improvement to bash autocompletion 2010-08-22 20:10:11 -03:00
profiling/priorityqueue mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
scrapy Refactoring of Crawler, Commands, Execution Queue and Spider Manager: 2010-08-25 19:24:36 -03:00
scripts removed python2.5 from rpm-install.sh script 2009-06-16 13:14:40 -03:00
.hgignore ignore docs/build 2009-07-25 15:21:22 -03:00
.hgtags Added tag 0.9 for changeset 5caf3dc10a92 2010-06-28 00:54:47 -03:00
AUTHORS Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in added some missing file to MANIFEST.in 2009-09-28 23:55:00 -03:00
README added scrapy-sqs.py to deployed scripts 2010-06-13 19:17:17 -03:00
setup.cfg added bitmap for windows installer 2009-09-17 02:01:40 -03:00
setup.py Removed scrapy-sqs script, as it has been superseded by the new scrapy 'queue' command 2010-08-23 22:04:49 -03:00

README 

This is Scrapy, an opensource screen scraping framework written in Python.

For more visit the project home page at http://scrapy.org