diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index e5c99e5b1..8c84b85fc 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -166,8 +166,7 @@ BaseItemExporter By default, this method looks for a serializer :ref:`declared in the item field ` and returns the result of applying that serializer to the value. If no serializer is found, it returns the - value unchanged except for ``unicode`` values which are encoded to - ``str`` using the encoding declared in the :attr:`encoding` attribute. + value unchanged. :param field: the field being serialized. If the source :ref:`item object ` does not define field metadata, *field* is an empty @@ -217,10 +216,7 @@ BaseItemExporter .. attribute:: encoding - The encoding that will be used to encode unicode values. This only - affects unicode values (which are always serialized to str using this - encoding). Other value types are passed unchanged to the specific - serialization library. + The output character encoding. .. attribute:: indent diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 29d9c5805..c0f534493 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -193,10 +193,10 @@ Item Loaders are declared using a class definition syntax. Here is an example:: default_output_processor = TakeFirst() - name_in = MapCompose(unicode.title) + name_in = MapCompose(str.title) name_out = Join() - price_in = MapCompose(unicode.strip) + price_in = MapCompose(str.strip) # ... diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index fbd8e4b73..1dffd1d55 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -51,12 +51,12 @@ Request objects given, the dict passed in this parameter will be shallow copied. :type meta: dict - :param body: the request body. If a ``unicode`` is passed, then it's encoded to - ``str`` using the ``encoding`` passed (which defaults to ``utf-8``). If - ``body`` is not given, an empty string is stored. Regardless of the - type of this argument, the final value stored will be a ``str`` (never - ``unicode`` or ``None``). - :type body: str or unicode + :param body: the request body. If a string is passed, then it's encoded as + bytes using the ``encoding`` passed (which defaults to ``utf-8``). If + ``body`` is not given, an empty bytes object is stored. Regardless of the + type of this argument, the final value stored will be a bytes object + (never a string or ``None``). + :type body: bytes or str :param headers: the headers of this request. The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If @@ -106,7 +106,7 @@ Request objects :param encoding: the encoding of this request (defaults to ``'utf-8'``). This encoding will be used to percent-encode the URL and to convert the - body to ``str`` (if given as ``unicode``). + body to bytes (if given as a string). :type encoding: string :param priority: the priority of this request (defaults to ``0``). @@ -721,7 +721,7 @@ Response objects .. attribute:: Response.body The body of this Response. Keep in mind that Response.body - is always a bytes object. If you want the unicode version use + is always a bytes object. If you want the string version use :attr:`TextResponse.text` (only available in :class:`TextResponse` and subclasses). @@ -842,9 +842,9 @@ TextResponse objects is the same as for the :class:`Response` class and is not documented here. :param encoding: is a string which contains the encoding to use for this - response. If you create a :class:`TextResponse` object with a unicode + response. If you create a :class:`TextResponse` object with a string as body, it will be encoded using this encoding (remember the body attribute - is always a string). If ``encoding`` is ``None`` (default value), the + is always a bytes object). If ``encoding`` is ``None`` (default value), the encoding will be looked up in the response headers and body instead. :type encoding: string @@ -853,7 +853,7 @@ TextResponse objects .. attribute:: TextResponse.text - Response body, as unicode. + Response body, as a string. The same as ``response.body.decode(response.encoding)``, but the result is cached after the first call, so you can access @@ -861,9 +861,11 @@ TextResponse objects .. note:: - ``unicode(response.body)`` is not a correct way to convert response - body to unicode: you would be using the system default encoding - (typically ``ascii``) instead of the response encoding. + ``str(response.body)`` is not a correct way to convert the response + body into a string: + + >>> str(b'body') + "b'body'" .. attribute:: TextResponse.encoding diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 5014df6ac..9e2c6ba42 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -64,7 +64,8 @@ more shortcuts: ``response.xpath()`` and ``response.css()``: Scrapy selectors are instances of :class:`~scrapy.selector.Selector` class constructed by passing either :class:`~scrapy.http.TextResponse` object or -markup as an unicode string (in ``text`` argument). +markup as a string (in ``text`` argument). + Usually there is no need to construct Scrapy selectors manually: ``response`` object is available in Spider callbacks, so in most cases it is more convenient to use ``response.css()`` and ``response.xpath()`` @@ -383,7 +384,7 @@ Using selectors with regular expressions :class:`~scrapy.selector.Selector` also has a ``.re()`` method for extracting data using regular expressions. However, unlike using ``.xpath()`` or -``.css()`` methods, ``.re()`` returns a list of unicode strings. So you +``.css()`` methods, ``.re()`` returns a list of strings. So you can't construct nested ``.re()`` calls. Here's an example used to extract image names from the :ref:`HTML code @@ -989,7 +990,7 @@ a :class:`~scrapy.http.HtmlResponse` object like this:: sel.xpath("//h1") 2. Extract the text of all ``

`` elements from an HTML response body, - returning a list of unicode strings:: + returning a list of strings:: sel.xpath("//h1").getall() # this includes the h1 tag sel.xpath("//h1/text()").getall() # this excludes the h1 tag