RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
2,687 Commits 42 Branches 132 Tags 168 MiB
Python 99.8%
HTML 0.1%
Go to file
Pablo Hoffman 409aaade0b Refactored close spider behaviour so that the engine now waits for all 
downloading (and enqueued for download) requests to finish and their responses
to be processed in the scraper/spiders, before closing the spider.

This will be required in the future to avoid loosing requests when we add
scheduler persistence and it's also a more correct behaviour overall.

The closing process has also been refactored to remove unneeded closing state
from downloader and leave it only in the engine.

Finally, some unused methods has been removed too, like spider_is_open() for
engine and scheduler.
 		2011-07-08 11:40:19 -03:00
bin slightly improved version of scrapyd script 2011-06-20 11:04:38 -03:00
debian debian/control: added python-setuptools to Recommends, because it's need by 'scrapy deploy' command 2011-05-01 11:00:02 -03:00
docs SitemapSpider: added support for filtering which sitemaps to follow (patch contributed by Rolando Espinoza). closes #330 2011-06-23 18:18:29 -03:00
extras added envvar SCRAPY_VERSION_FROM_HG=1 to extras/makedeb.py script 2011-06-18 03:31:47 -03:00
profiling/priorityqueue mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
scrapy Refactored close spider behaviour so that the engine now waits for all 2011-07-08 11:40:19 -03:00
scrapyd scrapyd activate_egg(): don't override SCRAPY_SETTINGS_MODULE envvar if already set 2011-06-15 19:35:03 -03:00
scripts removed python2.5 from rpm-install.sh script 2009-06-16 13:14:40 -03:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.hgignore ignore docs/build 2009-07-25 15:21:22 -03:00
.hgtags Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
AUTHORS added Darian Moody to AUTHORS 2011-06-12 01:42:30 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in added missing scrapyd/default_scrapyd.conf file to MANIFEST.in 2011-01-28 16:21:00 -02:00
Makefile.buildbot moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
README fixed typo 2011-04-07 02:04:42 -03:00
setup.cfg added bitmap for windows installer 2009-09-17 02:01:40 -03:00
setup.py added w3lib to requirements, in installation guide 2011-05-01 11:14:57 -03:00

README 

This is Scrapy, an opensource screen scraping framework written in Python.

For more info visit the project home page at http://scrapy.org