RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

updated FAQ entry to recommend using higher download delays

This commit is contained in:
Pablo Hoffman 2010-08-19 17:59:52 -03:00
parent 66525e77c7
commit 30e2404d8f
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/faq.rst
View File

@ -178,13 +178,14 @@ What does the response status code 999 means?
---------------------------------------------
999 is a custom reponse status code used by Yahoo sites to throttle requests.
Try slowing down the crawling speed by using a download delay in your spider::
Try slowing down the crawling speed by using a download delay of ``2`` (or
higher) in your spider::
class MySpider(CrawlSpider):
name = 'myspider'
download_delay = 1
download_delay = 2
# [ ... rest of the spider code ... ]