RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
2,733 Commits 42 Branches 132 Tags 165 MiB
Python 99.8%
HTML 0.1%
Go to file
Pablo Hoffman ce7a787970 Big downloader refactoring to support real concurrency limits per domain/ip, 
instead of global limits per spider which were a bit useless.

This removes the setting CONCURRENT_REQUESTS_PER_SPIDER and adds thre new
settings:

* CONCURRENT_REQUESTS
* CONCURRENT_REQUESTS_PER_DOMAIN
* CONCURRENT_REQUESTS_PER_IP (overrides per domain)

The AutoThrottle extension had to be disabled, but will be ported and
re-enabled soon.
 		2011-07-27 13:38:09 -03:00
bin slightly improved version of scrapyd script 2011-06-20 11:04:38 -03:00
debian debian/control: added python-setuptools to Recommends, because it's need by 'scrapy deploy' command 2011-05-01 11:00:02 -03:00
docs Big downloader refactoring to support real concurrency limits per domain/ip, 2011-07-27 13:38:09 -03:00
extras added envvar SCRAPY_VERSION_FROM_HG=1 to extras/makedeb.py script 2011-06-18 03:31:47 -03:00
profiling/priorityqueue mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
scrapy Big downloader refactoring to support real concurrency limits per domain/ip, 2011-07-27 13:38:09 -03:00
scrapyd moved spider queues to scrapyd 2011-07-19 19:39:27 -03:00
scripts removed python2.5 from rpm-install.sh script 2009-06-16 13:14:40 -03:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.hgignore ignore docs/build 2009-07-25 15:21:22 -03:00
.hgtags Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
AUTHORS added Darian Moody to AUTHORS 2011-06-12 01:42:30 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in added missing scrapyd/default_scrapyd.conf file to MANIFEST.in 2011-01-28 16:21:00 -02:00
Makefile.buildbot moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
README fixed typo 2011-04-07 02:04:42 -03:00
setup.cfg added bitmap for windows installer 2009-09-17 02:01:40 -03:00
setup.py setup.py: only add lxml requirement if libxml2 is not available 2011-07-13 13:04:42 -03:00

README 

This is Scrapy, an opensource screen scraping framework written in Python.

For more info visit the project home page at http://scrapy.org