RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Scrapy, a fast high-level web crawling & scraping framework for Python.
2,935 Commits 49 Branches 132 Tags 143 MiB
Python 99.8%
HTML 0.1%
Go to file
olveyra b39cb22d83 dont discard slot when empty, just save in another dict in order to recycle if needed again. 
This fix avoids to continuosly create new slot under certain cases, bug that prevents download_delay and max_concurrent_requests to work properly.

The problem arises when the slot for a given domain becomes empty, but further requests for that domain werent still created by the spider. This is typical when spider creates requests one by one, or it makes requests to multiple domains and one or more of them are created in a rate enough slow that makes slot to be empty each time the response is fetched.

The effect is that a new slot is created for each request under such conditions, and so the download_delay and max_concurrent_requests are not taking effect (because in order to apply, depends on an already existing slot for that domain).
 		2012-04-02 20:34:57 +00:00
bin slightly improved version of scrapyd script 2011-06-20 11:04:38 -03:00
debian bumped required python-w3lib version in debian/control 2012-03-05 20:25:38 -02:00
docs Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00
extras Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
scrapy dont discard slot when empty, just save in another dict in order to recycle if needed again. 2012-04-02 20:34:57 +00:00
scrapyd Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
.coveragerc Added rules to Makefile.buildbot for generating coverage reports 2010-12-15 11:13:45 -02:00
.gitignore Merge branch '0.12' 2011-09-18 01:08:04 -03:00
.hgtags Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
AUTHORS added Darian Moody to AUTHORS 2011-06-12 01:42:30 -03:00
INSTALL mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
LICENSE mv scrapy/trunk to root as part of svn2hg migration 2009-05-06 15:55:17 -03:00
MANIFEST.in moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
Makefile.buildbot moved coverage report script to extras/ 2010-12-15 12:08:59 -02:00
README fixed typo 2011-04-07 02:04:42 -03:00
setup.cfg moved rpm-install.sh to extras/ 2011-09-18 06:08:28 -03:00
setup.py Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00

README 

This is Scrapy, an opensource screen scraping framework written in Python.

For more info visit the project home page at http://scrapy.org