diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 835af2e74..a3647fd9b 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -603,10 +603,12 @@ HttpProxyMiddleware .. versionadded:: 0.8 +.. reqmeta:: proxy + .. class:: HttpProxyMiddleware This middleware sets the HTTP proxy to use for requests, by setting the - ``proxy`` meta value to :class:`~scrapy.http.Request` objects. + ``proxy`` meta value for :class:`~scrapy.http.Request` objects. Like the Python standard library modules `urllib`_ and `urllib2`_, it obeys the following environment variables: @@ -615,6 +617,9 @@ HttpProxyMiddleware * ``https_proxy`` * ``no_proxy`` + You can also set the meta key ``proxy`` per-request, to a value like + ``http://some_proxy_server:port``. + .. _urllib: http://docs.python.org/library/urllib.html .. _urllib2: http://docs.python.org/library/urllib2.html diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 68d87d04f..f2c37c2e6 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -224,12 +224,15 @@ Those are: * :reqmeta:`dont_redirect` * :reqmeta:`dont_retry` * :reqmeta:`handle_httpstatus_list` +* :reqmeta:`handle_httpstatus_all` * ``dont_merge_cookies`` (see ``cookies`` parameter of :class:`Request` constructor) * :reqmeta:`cookiejar` * :reqmeta:`redirect_urls` * :reqmeta:`bindaddress` * :reqmeta:`dont_obey_robotstxt` * :reqmeta:`download_timeout` +* :reqmeta:`download_maxsize` +* :reqmeta:`proxy` .. reqmeta:: bindaddress diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 0e9e53de4..ea2c208bb 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -433,6 +433,8 @@ The maximum response size (in bytes) that downloader will download. If you want to disable it set to 0. +.. reqmeta:: download_maxsize + .. note:: This size can be set per spider using :attr:`download_maxsize` diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 173c4a80c..79cfbcf5d 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -225,9 +225,12 @@ this:: .. reqmeta:: handle_httpstatus_list +.. reqmeta:: handle_httpstatus_all + The ``handle_httpstatus_list`` key of :attr:`Request.meta ` can also be used to specify which response codes to -allow on a per-request basis. +allow on a per-request basis. You can also set the meta key ``handle_httpstatus_all`` +to ``True`` if you want to allow any response code for a request. Keep in mind, however, that it's usually a bad idea to handle non-200 responses, unless you really know what you're doing.