RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
2,868 Commits 42 Branches 132 Tags 168 MiB
Python 99.8%
HTML 0.1%
Go to file
olveyra 8e9f607747 dont discard slot when empty, just save in another dict in order to recycle if needed again. 
This fix avoids to continuosly create new slot under certain cases, bug that prevents download_delay and max_concurrent_requests to work properly.

The problem arises when the slot for a given domain becomes empty, but further requests for that domain werent still created by the spider. This is typical when spider creates requests one by one, or it makes requests to multiple domains and one or more of them are created in a rate enough slow that makes slot to be empty each time the response is fetched.

The effect is that a new slot is created for each request under such conditions, and so the download_delay and max_concurrent_requests are not taking effect (because in order to apply, depends on an already existing slot for that domain).
 		2012-04-09 01:10:38 -03:00
bin slightly improved version of scrapyd script 2011-06-20 11:04:38 -03:00
debian updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
docs fixed minor mistake in Request objects documentation 2012-03-20 23:50:21 -03:00
extras extras/makedeb.py: no longer obtaining version from git 2012-01-02 13:28:24 -02:00
scrapy dont discard slot when empty, just save in another dict in order to recycle if needed again. 2012-04-09 01:10:38 -03:00
scrapyd removed some obsolete remaining code related to sqlite support in scrapy 2012-03-16 12:04:41 -03:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.gitignore Merge branch '0.12' 2011-09-18 01:08:04 -03:00
.hgtags Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
AUTHORS added Darian Moody to AUTHORS 2011-06-12 01:42:30 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
Makefile.buildbot moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
README fixed typo 2011-04-07 02:04:42 -03:00
setup.cfg moved rpm-install.sh to extras/ 2011-09-18 06:08:28 -03:00
setup.py added support for generating version from git revision 2011-09-11 11:24:12 -03:00

README 

This is Scrapy, an opensource screen scraping framework written in Python.

For more info visit the project home page at http://scrapy.org