mirror of https://github.com/scrapy/scrapy.git
updated request/response reference doc
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40738
This commit is contained in:
Pablo Hoffman
parent
da6a24b662
commit
5dc1e7e5ca
|
|
@ -10,21 +10,57 @@ Request and Response objects
|
|||
Quick overview
|
||||
==============
|
||||
|
||||
Scrapy uses requests and response objects for crawling web sites.
|
||||
Scrapy uses :class:`Request` and :class:`Response` objects for crawling web
|
||||
sites.
|
||||
|
||||
Typically, :class:`Request` objects are generated is the spiders and pass
|
||||
across the system until they reach the downloader which ends up performing the
|
||||
requests and downloading a HTTP url, to finally generate a :class:`Response`
|
||||
object that returns to the spider which generated the request.
|
||||
Typically, :class:`Request` objects are generated in the spiders and pass
|
||||
across the system until they reach the Downloader, which executes the request
|
||||
and returns a :class:`Response` object which goes back to the spider that
|
||||
generated the request.
|
||||
|
||||
Request objects
|
||||
===============
|
||||
|
||||
.. class:: Request
|
||||
.. class:: Request(url, callback=None, context=None, method='GET', body=None, headers=None, cookies=None, url_encoding='utf-8', dont_filter=None)
|
||||
|
||||
A :class:`Request` object represents an HTTP request, which is usually
|
||||
generated in the Spider and executed by the Downloader, and thus generating
|
||||
a :class:`Response`.
|
||||
|
||||
``url`` is a string containing the URL for this request
|
||||
|
||||
``callback`` is a function that will be called with the response of this
|
||||
request (once its downloaded) as its first parameter
|
||||
|
||||
``context`` can be a dict which will be accessible in the callback function
|
||||
in ``response.request.context`` in the callback function
|
||||
|
||||
``method`` is a string with the HTTP method of this request
|
||||
|
||||
``body`` is a string containing the request body or None if the request
|
||||
doesn't contain a body (ex. GET requests)
|
||||
|
||||
``headers`` is a multi-valued dict containing the headers of this request
|
||||
|
||||
``cookies`` is a dict containing the request cookies
|
||||
|
||||
``url_encoding`` is a string with the encoding of the url of this request.
|
||||
The request URL will be percent encoded using this encoding before
|
||||
downloading
|
||||
|
||||
``dont_filter`` is a boolean which indicates that this request should not
|
||||
be filtered by the scheduler. This is used when you want to perform an
|
||||
identical request multiple times, for whatever reason
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: Request.url
|
||||
|
||||
A string containing the URL of this request. Keep in mind that this
|
||||
attribute contains the escaped URL, so it can differ from the URL passed in
|
||||
the constructor.
|
||||
|
||||
.. attribute:: Request.method
|
||||
|
||||
A string representing the HTTP method in the request. This is guaranteed to
|
||||
|
|
@ -60,83 +96,108 @@ Attributes
|
|||
Unlike the ``meta`` attribute, this dict is not copied at all when the
|
||||
request is cloned using the ``copy()`` or ``replace()`` methods.
|
||||
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: Request.__init__(url, callback=None, context=None, method='GET', body=None, headers=None, cookies=None, url_encoding='utf-8', dont_filter=None)
|
||||
.. method:: Request.copy()
|
||||
|
||||
Instantiates a ``Request`` object with the given arguments:
|
||||
Return a new Request which is a copy of this Request. The attribute
|
||||
:attr:`Request.meta` is copied, while :attr:`Request.cache` is not.
|
||||
|
||||
``url`` is a string containing the URL for this request
|
||||
.. method:: Request.replace()
|
||||
|
||||
``callback`` is a function that will be called with the response of this
|
||||
request (once its downloaded) as its first parameter
|
||||
Return a Request object with the same members, except for those members
|
||||
given new values by whichever keyword arguments are specified. The attribute
|
||||
:attr:`Request.meta` is copied, while :attr:`Request.cache` is not.
|
||||
|
||||
``context`` can be a dict which will be accessible in the callback function
|
||||
in ``response.request.context`` in the callback function
|
||||
.. method:: Reponse.to_string()
|
||||
|
||||
``method`` is a string with the HTTP method of this request
|
||||
|
||||
``body`` is a string containing the request body or None if the request
|
||||
doesn't contain a body (ex. GET requests)
|
||||
|
||||
``headers`` is a multi-valued dict containing the headers of this request
|
||||
|
||||
``cookies`` is dict of the request cookies
|
||||
|
||||
``url_encoding`` is a string with the encoding of the url of this request.
|
||||
The request URL will be percent encoded using this encoding before
|
||||
downloading
|
||||
|
||||
``dont_filter`` is a boolean which indicates that this request should not
|
||||
be filtered by the scheduler. This is used when you want to perform an
|
||||
identical request multiple times, for whatever reason
|
||||
|
||||
.. class:: Response
|
||||
Return a string with the raw HTTP representation of this response.
|
||||
|
||||
Response objects
|
||||
================
|
||||
|
||||
Attributes
|
||||
----------
|
||||
.. class:: Response(domain, url, status=200, headers=None, body=None)
|
||||
|
||||
.. attribute:: Response.status
|
||||
|
||||
An integer representing the HTTP status in the response. Example: ``200``,
|
||||
``404``, etc
|
||||
|
||||
.. attribute:: Response.headers
|
||||
|
||||
A dictionary-like object which contains the response headers.
|
||||
|
||||
.. attribute:: Response.meta
|
||||
|
||||
A dict that contains arbitrary metadata fro this response. It works like
|
||||
:attr:`Request.meta` for Request objects. See that attribute help for more
|
||||
info.
|
||||
|
||||
.. attribute:: Response.cache
|
||||
|
||||
A dict that contains arbitrary cached data for this response. It works like
|
||||
:attr:`Request.cache` for Request objects. See that attribute help for more
|
||||
info.
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: __init__(domain, url, original_url=None, headers=None, status=200, body=None)
|
||||
|
||||
Instantiates a ``Response`` object with the given arguments:
|
||||
A :class:`Response` object represents an HTTP response, which is usually
|
||||
downloaded (by the Downloader) and fed to the Spiders for processing.
|
||||
|
||||
``domain`` is a string with the domain of the spider for which this
|
||||
Response is for
|
||||
|
||||
``url`` is a string containing the URL for this response
|
||||
|
||||
``original_url`` is a string containing the url from which this response
|
||||
was redirected (only for redirected responses)
|
||||
|
||||
``headers`` is a multivalued dict of the response headers
|
||||
|
||||
``status`` is an integer with the HTTP status of the response
|
||||
|
||||
``body`` is a string (or unicode) containing the response body
|
||||
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: Response.url
|
||||
|
||||
A string containing the URL of the reponse.
|
||||
|
||||
.. attribute:: Response.status
|
||||
|
||||
An integer representing the HTTP status of the response. Example: ``200``,
|
||||
``404``.
|
||||
|
||||
.. attribute:: Response.headers
|
||||
|
||||
A dictionary-like object which contains the response headers.
|
||||
|
||||
.. attribute:: Response.body
|
||||
|
||||
The body of this Response.
|
||||
|
||||
.. attribute:: Response.request
|
||||
|
||||
The :class:`Request` object that generated this response. This attribute is
|
||||
assigned in the Scrapy engine, after the response and request has passed
|
||||
through all :ref:`Downloader Middlewares <topics-downloader-middleware>`.
|
||||
In particular, this means that:
|
||||
|
||||
- HTTP redirections will cause the original request (to the URL before
|
||||
redirection) to be assigned to the redirected response (with the final
|
||||
URL after redirection).
|
||||
|
||||
- Response.request.url doesn't always equals Response.url
|
||||
|
||||
- This attribute is only available in the spider code, and in the
|
||||
:ref:`Spider Middlewares <topics-spider-middleware>`, but not in
|
||||
Downloader Middlewares (although you have the Request available there by
|
||||
other means) and handlers of the :signal:`response_downloaded` signal.
|
||||
|
||||
.. attribute:: Response.meta
|
||||
|
||||
A dict that contains arbitrary metadata for this response, similar to the
|
||||
:attr:`Request.meta` attribute. See the :attr:`Request.meta` attribute for
|
||||
more info.
|
||||
|
||||
.. attribute:: Response.cache
|
||||
|
||||
A dict that contains arbitrary cached data for this response, similar to
|
||||
the :attr:`Request.cache` attribute. See the :attr:`Request.cache`
|
||||
attribute for more info.
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: Response.copy()
|
||||
|
||||
Return a new Response which is a copy of this Response. The attribute
|
||||
:attr:`Response.meta` is copied, while :attr:`Response.cache` is not.
|
||||
|
||||
.. method:: Response.replace(domain=None, url=None, status=None, headers=None, body=None)
|
||||
|
||||
Return a Response object with the same members, except for those members
|
||||
given new values by whichever keyword arguments are specified. The attribute
|
||||
:attr:`Response.meta` is copied, while :attr:`Response.cache` is not.
|
||||
|
||||
.. method:: Reponse.to_string()
|
||||
|
||||
Return a string with the raw HTTP representation of this response.
|
||||
|
|
|
|||
Loading…
Reference in New Issue