diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 835af2e74..fbc5391fa 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -603,6 +603,8 @@ HttpProxyMiddleware .. versionadded:: 0.8 +.. reqmeta:: proxy + .. class:: HttpProxyMiddleware This middleware sets the HTTP proxy to use for requests, by setting the diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 68d87d04f..f2c37c2e6 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -224,12 +224,15 @@ Those are: * :reqmeta:`dont_redirect` * :reqmeta:`dont_retry` * :reqmeta:`handle_httpstatus_list` +* :reqmeta:`handle_httpstatus_all` * ``dont_merge_cookies`` (see ``cookies`` parameter of :class:`Request` constructor) * :reqmeta:`cookiejar` * :reqmeta:`redirect_urls` * :reqmeta:`bindaddress` * :reqmeta:`dont_obey_robotstxt` * :reqmeta:`download_timeout` +* :reqmeta:`download_maxsize` +* :reqmeta:`proxy` .. reqmeta:: bindaddress diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 0e9e53de4..ea2c208bb 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -433,6 +433,8 @@ The maximum response size (in bytes) that downloader will download. If you want to disable it set to 0. +.. reqmeta:: download_maxsize + .. note:: This size can be set per spider using :attr:`download_maxsize` diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 92dc6ac47..fc707a87a 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -225,9 +225,12 @@ this:: .. reqmeta:: handle_httpstatus_list +.. reqmeta:: handle_httpstatus_all + The ``handle_httpstatus_list`` key of :attr:`Request.meta ` can also be used to specify which response codes to -allow on a per-request basis. +allow on a per-request basis. You can also set the meta key ``handle_httpstatus_all`` +to ``True`` if you want to allow any response code for a request. Keep in mind, however, that it's usually a bad idea to handle non-200 responses, unless you really know what you're doing.