Merge pull request #1530 from scrapy/retry-400

[MRG+1] DOC fix docs after GH-1289.

docs/topics/downloader-middleware.rst

@@ -819,12 +819,6 @@ settings (see the settings documentation for more info):
 * :setting:`RETRY_TIMES`
 * :setting:`RETRY_HTTP_CODES`
 
-About HTTP errors to consider:
-
-You may want to remove 400 from :setting:`RETRY_HTTP_CODES`, if you stick to the
-HTTP protocol. It's included by default because it's a common code used
-to indicate server overload, which would be something we want to retry.
-
 .. reqmeta:: dont_retry
 
 If :attr:`Request.meta <scrapy.http.Request.meta>` has ``dont_retry`` key
@@ -858,11 +852,16 @@ Maximum number of times to retry, in addition to the first download.
 RETRY_HTTP_CODES
 ^^^^^^^^^^^^^^^^
 
-Default: ``[500, 502, 503, 504, 400, 408]``
+Default: ``[500, 502, 503, 504, 408]``
 
 Which HTTP response codes to retry. Other errors (DNS lookup issues,
 connections lost, etc) are always retried.
 
+In some cases you may want to add 400 to :setting:`RETRY_HTTP_CODES` because
+it is a common code used to indicate server overload. It is not included by
+default because HTTP specs say so.
+
+
 .. _topics-dlmw-robots:
 
 RobotsTxtMiddleware

scrapy/downloadermiddlewares/retry.py

@@ -10,12 +10,6 @@ Failed pages are collected on the scraping process and rescheduled at the end,
 once the spider has finished crawling all regular (non failed) pages. Once
 there is no more failed pages to retry this middleware sends a signal
 (retry_complete), so other extensions could connect to that signal.
-
-About HTTP errors to consider:
-
-- You may want to remove 400 from RETRY_HTTP_CODES, if you stick to the HTTP
-  protocol. It's included by default because it's a common code used to
-  indicate server overload, which would be something we want to retry
 """
 import logging