From 30e2404d8fb835452c506c87308d38f71b4ff438 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Thu, 19 Aug 2010 17:59:52 -0300 Subject: [PATCH] updated FAQ entry to recommend using higher download delays --- docs/faq.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index ea301563d..3f837760b 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -178,13 +178,14 @@ What does the response status code 999 means? --------------------------------------------- 999 is a custom reponse status code used by Yahoo sites to throttle requests. -Try slowing down the crawling speed by using a download delay in your spider:: +Try slowing down the crawling speed by using a download delay of ``2`` (or +higher) in your spider:: class MySpider(CrawlSpider): name = 'myspider' - download_delay = 1 + download_delay = 2 # [ ... rest of the spider code ... ]