mirror of https://github.com/scrapy/scrapy.git
Enable doc tests for selectors.rst, fix issues.
This commit is contained in:
Andrey Rakhmatullin
parent
776cf59990
commit
cba891a66c
|
|
@ -48,6 +48,8 @@ Constructing selectors
|
|||
|
||||
.. highlight:: python
|
||||
|
||||
.. skip: start
|
||||
|
||||
Response objects expose a :class:`~scrapy.Selector` instance
|
||||
on ``.selector`` attribute:
|
||||
|
||||
|
|
@ -66,6 +68,8 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
|
|||
>>> response.css("span::text").get()
|
||||
'good'
|
||||
|
||||
.. skip: end
|
||||
|
||||
Scrapy selectors are instances of :class:`~scrapy.Selector` class
|
||||
constructed by passing either :class:`~scrapy.http.TextResponse` object or
|
||||
markup as a string (in ``text`` argument).
|
||||
|
|
@ -93,7 +97,7 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
|
|||
|
||||
>>> from scrapy.selector import Selector
|
||||
>>> from scrapy.http import HtmlResponse
|
||||
>>> response = HtmlResponse(url="http://example.com", body=body)
|
||||
>>> response = HtmlResponse(url="http://example.com", body=body, encoding="utf-8")
|
||||
>>> Selector(response=response).xpath("//span/text()").get()
|
||||
'good'
|
||||
|
||||
|
|
@ -103,6 +107,13 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
|
|||
Using selectors
|
||||
---------------
|
||||
|
||||
.. invisible-code-block: python
|
||||
|
||||
html_response = response = load_response(
|
||||
"https://docs.scrapy.org/en/latest/_static/selectors-sample1.html",
|
||||
"../_static/selectors-sample1.html",
|
||||
)
|
||||
|
||||
To explain how to use the selectors we'll use the ``Scrapy shell`` (which
|
||||
provides interactive testing) and an example page located in the Scrapy
|
||||
documentation server:
|
||||
|
|
@ -135,7 +146,7 @@ page, let's construct an XPath for selecting the text inside the title tag:
|
|||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath("//title/text()")
|
||||
[<Selector xpath='//title/text()' data='Example website'>]
|
||||
[<Selector query='//title/text()' data='Example website'>]
|
||||
|
||||
To actually extract the textual data, you must call the selector ``.get()``
|
||||
or ``.getall()`` methods, as follows:
|
||||
|
|
@ -363,11 +374,11 @@ too. Here's an example:
|
|||
|
||||
>>> links = response.xpath('//a[contains(@href, "image")]')
|
||||
>>> links.getall()
|
||||
['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
|
||||
'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
|
||||
'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
|
||||
'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
|
||||
'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
|
||||
['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg" alt="image1"></a>',
|
||||
'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg" alt="image2"></a>',
|
||||
'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg" alt="image3"></a>',
|
||||
'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg" alt="image4"></a>',
|
||||
'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg" alt="image5"></a>']
|
||||
|
||||
>>> for index, link in enumerate(links):
|
||||
... href_xpath = link.xpath("@href").get()
|
||||
|
|
@ -447,11 +458,11 @@ Here's an example used to extract image names from the :ref:`HTML code
|
|||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
|
||||
['My image 1',
|
||||
'My image 2',
|
||||
'My image 3',
|
||||
'My image 4',
|
||||
'My image 5']
|
||||
['My image 1 ',
|
||||
'My image 2 ',
|
||||
'My image 3 ',
|
||||
'My image 4 ',
|
||||
'My image 5 ']
|
||||
|
||||
There's an additional helper reciprocating ``.get()`` (and its
|
||||
alias ``.extract_first()``) for ``.re()``, named ``.re_first()``.
|
||||
|
|
@ -460,7 +471,7 @@ Use it to extract just the first matching string:
|
|||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
|
||||
'My image 1'
|
||||
'My image 1 '
|
||||
|
||||
.. _old-extraction-api:
|
||||
|
||||
|
|
@ -761,6 +772,8 @@ on `XPath variables`_.
|
|||
Removing namespaces
|
||||
-------------------
|
||||
|
||||
.. skip: start
|
||||
|
||||
When dealing with scraping projects, it is often quite convenient to get rid of
|
||||
namespaces altogether and just work with element names, to write more
|
||||
simple/convenient XPaths. You can use the
|
||||
|
|
@ -808,8 +821,8 @@ nodes can be accessed directly by their names:
|
|||
|
||||
>>> response.selector.remove_namespaces()
|
||||
>>> response.xpath("//link")
|
||||
[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
|
||||
<Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
|
||||
[<Selector query='//link' data='<link rel="alternate" type="text/html" h'>,
|
||||
<Selector query='//link' data='<link rel="next" type="application/atom+'>,
|
||||
...
|
||||
|
||||
If you wonder why the namespace removal procedure isn't always called by default
|
||||
|
|
@ -824,6 +837,7 @@ of relevance, are:
|
|||
case some element names clash between namespaces. These cases are very rare
|
||||
though.
|
||||
|
||||
.. skip: end
|
||||
|
||||
Using EXSLT extensions
|
||||
----------------------
|
||||
|
|
@ -881,6 +895,8 @@ extracting text elements for example.
|
|||
Example extracting microdata (sample content taken from https://schema.org/Product)
|
||||
with groups of itemscopes and corresponding itemprops:
|
||||
|
||||
.. skip: next
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> doc = """
|
||||
|
|
@ -977,26 +993,35 @@ Scrapy selectors also provide a sorely missed XPath extension function
|
|||
``has-class`` that returns ``True`` for nodes that have all of the specified
|
||||
HTML classes.
|
||||
|
||||
.. highlight:: html
|
||||
For the following HTML:
|
||||
|
||||
For the following HTML::
|
||||
.. code-block:: pycon
|
||||
|
||||
<p class="foo bar-baz">First</p>
|
||||
<p class="foo">Second</p>
|
||||
<p class="bar">Third</p>
|
||||
<p>Fourth</p>
|
||||
|
||||
.. highlight:: python
|
||||
>>> from scrapy.http import HtmlResponse
|
||||
>>> response = HtmlResponse(
|
||||
... url="http://example.com",
|
||||
... body="""
|
||||
... <html>
|
||||
... <body>
|
||||
... <p class="foo bar-baz">First</p>
|
||||
... <p class="foo">Second</p>
|
||||
... <p class="bar">Third</p>
|
||||
... <p>Fourth</p>
|
||||
... </body>
|
||||
... </html>
|
||||
... """,
|
||||
... encoding="utf-8",
|
||||
... )
|
||||
|
||||
You can use it like this:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.xpath('//p[has-class("foo")]')
|
||||
[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
|
||||
<Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
|
||||
[<Selector query='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
|
||||
<Selector query='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
|
||||
[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
|
||||
[<Selector query='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
|
||||
>>> response.xpath('//p[has-class("foo", "bar")]')
|
||||
[]
|
||||
|
||||
|
|
@ -1132,6 +1157,8 @@ a :class:`~scrapy.http.HtmlResponse` object like this:
|
|||
Selector examples on XML response
|
||||
---------------------------------
|
||||
|
||||
.. skip: start
|
||||
|
||||
Here are some examples to illustrate concepts for :class:`Selector` objects
|
||||
instantiated with an :class:`~scrapy.http.XmlResponse` object:
|
||||
|
||||
|
|
@ -1154,4 +1181,6 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:
|
|||
sel.register_namespace("g", "http://base.google.com/ns/1.0")
|
||||
sel.xpath("//g:price").getall()
|
||||
|
||||
.. skip: end
|
||||
|
||||
.. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799
|
||||
|
|
|
|||
Loading…
Reference in New Issue