Scrapy, a fast high-level web crawling & scraping framework for Python.
Go to file
Pablo Hoffman 3c919f2562 Several core changes:
Execution Manager:

* added control_reactor argument to delegate external twisted
  reactor control (for example by twistd)
* now it loads spiders (if not already loaded)
* now it stars the log (if not already started)
* removed *args from configure() method
* removed **opts from runonce and start methods

Execution engine:

* added control_reactor argument to to delegate external twisted
  reactor control (for example by twistd)
* changed some functions and method names for clarity
* improve handling of exceptions in st() method
* regrouped close_domain, closed_domain, and _close_domain method
  for legibilty

Scheduler:

* replaced pending_domains_count (dict) by pending_domains (set)
* simplified some doc
2009-06-15 19:40:56 -03:00
bin sorted out running of unittests: 2009-05-16 20:11:23 -03:00
docs added StatsMailer extension 2009-06-12 15:38:21 -03:00
examples Some changes to HTTP cache middleware: 2009-05-24 19:13:06 -03:00
extras/sql mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
profiling/priorityqueue mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
scrapy Several core changes: 2009-06-15 19:40:56 -03:00
scripts mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
.hgignore adding .svn to hgignore to help with hg2svn autocommits 2009-05-07 17:29:30 -03:00
AUTHORS mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
README mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
setup.cfg mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
setup.py mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00

README

This is Scrapy, an opensource screen scraping framework written in Python.

For more visit the project home page at http://scrapy.org