diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index c25c75d17..4a64d530b 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -48,6 +48,8 @@ Constructing selectors .. highlight:: python +.. skip: start + Response objects expose a :class:`~scrapy.Selector` instance on ``.selector`` attribute: @@ -66,6 +68,8 @@ more shortcuts: ``response.xpath()`` and ``response.css()``: >>> response.css("span::text").get() 'good' +.. skip: end + Scrapy selectors are instances of :class:`~scrapy.Selector` class constructed by passing either :class:`~scrapy.http.TextResponse` object or markup as a string (in ``text`` argument). @@ -93,7 +97,7 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of >>> from scrapy.selector import Selector >>> from scrapy.http import HtmlResponse - >>> response = HtmlResponse(url="http://example.com", body=body) + >>> response = HtmlResponse(url="http://example.com", body=body, encoding="utf-8") >>> Selector(response=response).xpath("//span/text()").get() 'good' @@ -103,6 +107,13 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of Using selectors --------------- +.. invisible-code-block: python + + html_response = response = load_response( + "https://docs.scrapy.org/en/latest/_static/selectors-sample1.html", + "../_static/selectors-sample1.html", + ) + To explain how to use the selectors we'll use the ``Scrapy shell`` (which provides interactive testing) and an example page located in the Scrapy documentation server: @@ -135,7 +146,7 @@ page, let's construct an XPath for selecting the text inside the title tag: .. code-block:: pycon >>> response.xpath("//title/text()") - [] + [] To actually extract the textual data, you must call the selector ``.get()`` or ``.getall()`` methods, as follows: @@ -363,11 +374,11 @@ too. Here's an example: >>> links = response.xpath('//a[contains(@href, "image")]') >>> links.getall() - ['Name: My image 1
', - 'Name: My image 2
', - 'Name: My image 3
', - 'Name: My image 4
', - 'Name: My image 5
'] + ['Name: My image 1
image1
', + 'Name: My image 2
image2
', + 'Name: My image 3
image3
', + 'Name: My image 4
image4
', + 'Name: My image 5
image5
'] >>> for index, link in enumerate(links): ... href_xpath = link.xpath("@href").get() @@ -447,11 +458,11 @@ Here's an example used to extract image names from the :ref:`HTML code .. code-block:: pycon >>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)") - ['My image 1', - 'My image 2', - 'My image 3', - 'My image 4', - 'My image 5'] + ['My image 1 ', + 'My image 2 ', + 'My image 3 ', + 'My image 4 ', + 'My image 5 '] There's an additional helper reciprocating ``.get()`` (and its alias ``.extract_first()``) for ``.re()``, named ``.re_first()``. @@ -460,7 +471,7 @@ Use it to extract just the first matching string: .. code-block:: pycon >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)") - 'My image 1' + 'My image 1 ' .. _old-extraction-api: @@ -761,6 +772,8 @@ on `XPath variables`_. Removing namespaces ------------------- +.. skip: start + When dealing with scraping projects, it is often quite convenient to get rid of namespaces altogether and just work with element names, to write more simple/convenient XPaths. You can use the @@ -808,8 +821,8 @@ nodes can be accessed directly by their names: >>> response.selector.remove_namespaces() >>> response.xpath("//link") - [, - , + [, + , ... If you wonder why the namespace removal procedure isn't always called by default @@ -824,6 +837,7 @@ of relevance, are: case some element names clash between namespaces. These cases are very rare though. +.. skip: end Using EXSLT extensions ---------------------- @@ -881,6 +895,8 @@ extracting text elements for example. Example extracting microdata (sample content taken from https://schema.org/Product) with groups of itemscopes and corresponding itemprops: +.. skip: next + .. code-block:: pycon >>> doc = """ @@ -977,26 +993,35 @@ Scrapy selectors also provide a sorely missed XPath extension function ``has-class`` that returns ``True`` for nodes that have all of the specified HTML classes. -.. highlight:: html +For the following HTML: -For the following HTML:: +.. code-block:: pycon -

First

-

Second

-

Third

-

Fourth

- -.. highlight:: python + >>> from scrapy.http import HtmlResponse + >>> response = HtmlResponse( + ... url="http://example.com", + ... body=""" + ... + ... + ...

First

+ ...

Second

+ ...

Third

+ ...

Fourth

+ ... + ... + ... """, + ... encoding="utf-8", + ... ) You can use it like this: .. code-block:: pycon >>> response.xpath('//p[has-class("foo")]') - [, - ] + [, + ] >>> response.xpath('//p[has-class("foo", "bar-baz")]') - [] + [] >>> response.xpath('//p[has-class("foo", "bar")]') [] @@ -1132,6 +1157,8 @@ a :class:`~scrapy.http.HtmlResponse` object like this: Selector examples on XML response --------------------------------- +.. skip: start + Here are some examples to illustrate concepts for :class:`Selector` objects instantiated with an :class:`~scrapy.http.XmlResponse` object: @@ -1154,4 +1181,6 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object: sel.register_namespace("g", "http://base.google.com/ns/1.0") sel.xpath("//g:price").getall() +.. skip: end + .. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799