From 1b6d60c2514989402cd1c94d5c0ea8594f5aef16 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Tue, 6 Oct 2015 23:56:28 +0500 Subject: [PATCH] DOC fix docs after GH-1289. --- docs/topics/downloader-middleware.rst | 13 ++++++------- scrapy/downloadermiddlewares/retry.py | 6 ------ 2 files changed, 6 insertions(+), 13 deletions(-) diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index b79da1b95..9122e5cb5 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -819,12 +819,6 @@ settings (see the settings documentation for more info): * :setting:`RETRY_TIMES` * :setting:`RETRY_HTTP_CODES` -About HTTP errors to consider: - -You may want to remove 400 from :setting:`RETRY_HTTP_CODES`, if you stick to the -HTTP protocol. It's included by default because it's a common code used -to indicate server overload, which would be something we want to retry. - .. reqmeta:: dont_retry If :attr:`Request.meta ` has ``dont_retry`` key @@ -858,11 +852,16 @@ Maximum number of times to retry, in addition to the first download. RETRY_HTTP_CODES ^^^^^^^^^^^^^^^^ -Default: ``[500, 502, 503, 504, 400, 408]`` +Default: ``[500, 502, 503, 504, 408]`` Which HTTP response codes to retry. Other errors (DNS lookup issues, connections lost, etc) are always retried. +In some cases you may want to add 400 to :setting:`RETRY_HTTP_CODES` because +it is a common code used to indicate server overload. It is not included by +default because HTTP specs say so. + + .. _topics-dlmw-robots: RobotsTxtMiddleware diff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.py index 714103ba0..803ed5fc0 100644 --- a/scrapy/downloadermiddlewares/retry.py +++ b/scrapy/downloadermiddlewares/retry.py @@ -10,12 +10,6 @@ Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages. Once there is no more failed pages to retry this middleware sends a signal (retry_complete), so other extensions could connect to that signal. - -About HTTP errors to consider: - -- You may want to remove 400 from RETRY_HTTP_CODES, if you stick to the HTTP - protocol. It's included by default because it's a common code used to - indicate server overload, which would be something we want to retry """ import logging