Documented handle_httpstatus_list request.meta key

This commit is contained in:
Pablo Hoffman 2010-09-09 21:50:40 -03:00
parent f1c943543a
commit 7f21a6384f
2 changed files with 20 additions and 13 deletions

View File

@ -219,6 +219,7 @@ Those are:
* :reqmeta:`dont_redirect`
* :reqmeta:`dont_retry`
* :reqmeta:`handle_httpstatus_list`
.. _topics-request-response-ref-request-subclasses:

View File

@ -174,28 +174,34 @@ HttpErrorMiddleware
.. class:: HttpErrorMiddleware
Filter out unsuccessful (erroneous) HTTP responses so that spiders don't
have to deal with them, which (most of the times) imposes an overhead,
have to deal with them, which (most of the time) imposes an overhead,
consumes more resources, and makes the spider logic more complex.
According to the `HTTP standard`_, successful responses are those whose
status codes are in the 200-300 range.
According to the `HTTP standard`_, successful responses are those whose
status codes are in the 200-300 range.
.. _HTTP standard: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
If you still want to process response codes outside that range, you can
specify which response codes the spider is able to handle using the
``handle_httpstatus_list`` spider attribute.
If you still want to process response codes outside that range, you can
specify which response codes the spider is able to handle using the
``handle_httpstatus_list`` spider attribute.
For example, if you want your spider to handle 404 responses you can do
this::
For example, if you want your spider to handle 404 responses you can do
this::
class MySpider(CrawlSpider):
handle_httpstatus_list = [404]
class MySpider(CrawlSpider):
handle_httpstatus_list = [404]
Keep in mind, however, that it's usually a bad idea to handle non-200
responses, unless you really know what you're doing.
.. reqmeta:: handle_httpstatus_list
For more information see: `HTTP Status Code Definitions`_.
The ``handle_httpstatus_list`` key of :attr:`Request.meta
<scrapy.http.Request.meta>` can also be used to specify which response codes to
allow on a per-request basis.
Keep in mind, however, that it's usually a bad idea to handle non-200
responses, unless you really know what you're doing.
For more information see: `HTTP Status Code Definitions`_.
.. _HTTP Status Code Definitions: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html