From 5dc1e7e5ca1d41d1956491c6e8c4818ddaad8460 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Sat, 17 Jan 2009 21:05:08 +0000 Subject: [PATCH] updated request/response reference doc --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40738 --- scrapy/trunk/docs/ref/request-response.rst | 189 ++++++++++++++------- 1 file changed, 125 insertions(+), 64 deletions(-) diff --git a/scrapy/trunk/docs/ref/request-response.rst b/scrapy/trunk/docs/ref/request-response.rst index 1b050f277..2cc033e2a 100644 --- a/scrapy/trunk/docs/ref/request-response.rst +++ b/scrapy/trunk/docs/ref/request-response.rst @@ -10,21 +10,57 @@ Request and Response objects Quick overview ============== -Scrapy uses requests and response objects for crawling web sites. +Scrapy uses :class:`Request` and :class:`Response` objects for crawling web +sites. -Typically, :class:`Request` objects are generated is the spiders and pass -across the system until they reach the downloader which ends up performing the -requests and downloading a HTTP url, to finally generate a :class:`Response` -object that returns to the spider which generated the request. +Typically, :class:`Request` objects are generated in the spiders and pass +across the system until they reach the Downloader, which executes the request +and returns a :class:`Response` object which goes back to the spider that +generated the request. Request objects =============== -.. class:: Request +.. class:: Request(url, callback=None, context=None, method='GET', body=None, headers=None, cookies=None, url_encoding='utf-8', dont_filter=None) + + A :class:`Request` object represents an HTTP request, which is usually + generated in the Spider and executed by the Downloader, and thus generating + a :class:`Response`. + + ``url`` is a string containing the URL for this request + + ``callback`` is a function that will be called with the response of this + request (once its downloaded) as its first parameter + + ``context`` can be a dict which will be accessible in the callback function + in ``response.request.context`` in the callback function + + ``method`` is a string with the HTTP method of this request + + ``body`` is a string containing the request body or None if the request + doesn't contain a body (ex. GET requests) + + ``headers`` is a multi-valued dict containing the headers of this request + + ``cookies`` is a dict containing the request cookies + + ``url_encoding`` is a string with the encoding of the url of this request. + The request URL will be percent encoded using this encoding before + downloading + + ``dont_filter`` is a boolean which indicates that this request should not + be filtered by the scheduler. This is used when you want to perform an + identical request multiple times, for whatever reason Attributes ---------- +.. attribute:: Request.url + + A string containing the URL of this request. Keep in mind that this + attribute contains the escaped URL, so it can differ from the URL passed in + the constructor. + .. attribute:: Request.method A string representing the HTTP method in the request. This is guaranteed to @@ -60,83 +96,108 @@ Attributes Unlike the ``meta`` attribute, this dict is not copied at all when the request is cloned using the ``copy()`` or ``replace()`` methods. - Methods ------- -.. method:: Request.__init__(url, callback=None, context=None, method='GET', body=None, headers=None, cookies=None, url_encoding='utf-8', dont_filter=None) +.. method:: Request.copy() - Instantiates a ``Request`` object with the given arguments: + Return a new Request which is a copy of this Request. The attribute + :attr:`Request.meta` is copied, while :attr:`Request.cache` is not. - ``url`` is a string containing the URL for this request +.. method:: Request.replace() - ``callback`` is a function that will be called with the response of this - request (once its downloaded) as its first parameter + Return a Request object with the same members, except for those members + given new values by whichever keyword arguments are specified. The attribute + :attr:`Request.meta` is copied, while :attr:`Request.cache` is not. - ``context`` can be a dict which will be accessible in the callback function - in ``response.request.context`` in the callback function +.. method:: Reponse.to_string() - ``method`` is a string with the HTTP method of this request - - ``body`` is a string containing the request body or None if the request - doesn't contain a body (ex. GET requests) - - ``headers`` is a multi-valued dict containing the headers of this request - - ``cookies`` is dict of the request cookies - - ``url_encoding`` is a string with the encoding of the url of this request. - The request URL will be percent encoded using this encoding before - downloading - - ``dont_filter`` is a boolean which indicates that this request should not - be filtered by the scheduler. This is used when you want to perform an - identical request multiple times, for whatever reason - -.. class:: Response + Return a string with the raw HTTP representation of this response. Response objects ================ -Attributes ----------- +.. class:: Response(domain, url, status=200, headers=None, body=None) -.. attribute:: Response.status - - An integer representing the HTTP status in the response. Example: ``200``, - ``404``, etc - -.. attribute:: Response.headers - - A dictionary-like object which contains the response headers. - -.. attribute:: Response.meta - - A dict that contains arbitrary metadata fro this response. It works like - :attr:`Request.meta` for Request objects. See that attribute help for more - info. - -.. attribute:: Response.cache - - A dict that contains arbitrary cached data for this response. It works like - :attr:`Request.cache` for Request objects. See that attribute help for more - info. - -Methods -------- - -.. method:: __init__(domain, url, original_url=None, headers=None, status=200, body=None) - - Instantiates a ``Response`` object with the given arguments: + A :class:`Response` object represents an HTTP response, which is usually + downloaded (by the Downloader) and fed to the Spiders for processing. + + ``domain`` is a string with the domain of the spider for which this + Response is for ``url`` is a string containing the URL for this response - ``original_url`` is a string containing the url from which this response - was redirected (only for redirected responses) - ``headers`` is a multivalued dict of the response headers ``status`` is an integer with the HTTP status of the response ``body`` is a string (or unicode) containing the response body + +Attributes +---------- + +.. attribute:: Response.url + + A string containing the URL of the reponse. + +.. attribute:: Response.status + + An integer representing the HTTP status of the response. Example: ``200``, + ``404``. + +.. attribute:: Response.headers + + A dictionary-like object which contains the response headers. + +.. attribute:: Response.body + + The body of this Response. + +.. attribute:: Response.request + + The :class:`Request` object that generated this response. This attribute is + assigned in the Scrapy engine, after the response and request has passed + through all :ref:`Downloader Middlewares `. + In particular, this means that: + + - HTTP redirections will cause the original request (to the URL before + redirection) to be assigned to the redirected response (with the final + URL after redirection). + + - Response.request.url doesn't always equals Response.url + + - This attribute is only available in the spider code, and in the + :ref:`Spider Middlewares `, but not in + Downloader Middlewares (although you have the Request available there by + other means) and handlers of the :signal:`response_downloaded` signal. + +.. attribute:: Response.meta + + A dict that contains arbitrary metadata for this response, similar to the + :attr:`Request.meta` attribute. See the :attr:`Request.meta` attribute for + more info. + +.. attribute:: Response.cache + + A dict that contains arbitrary cached data for this response, similar to + the :attr:`Request.cache` attribute. See the :attr:`Request.cache` + attribute for more info. + +Methods +------- + +.. method:: Response.copy() + + Return a new Response which is a copy of this Response. The attribute + :attr:`Response.meta` is copied, while :attr:`Response.cache` is not. + +.. method:: Response.replace(domain=None, url=None, status=None, headers=None, body=None) + + Return a Response object with the same members, except for those members + given new values by whichever keyword arguments are specified. The attribute + :attr:`Response.meta` is copied, while :attr:`Response.cache` is not. + +.. method:: Reponse.to_string() + + Return a string with the raw HTTP representation of this response.