RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/ref/selectors.rst

6.3 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

XPath Selectors API

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "module".

.. module:: scrapy.xpath
   :synopsis: XPath selectors classes

There are two types of selectors bundled with Scrapy: :class:`HtmlXPathSelector` and :class:`XmlXPathSelector`. Both of them implement the same :class:`XPathSelector` interface. The only different is that one is used to process HTML data and the other XML data.

System Message: ERROR/3 (<stdin>, line 10); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 10); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 10); backlink

Unknown interpreted text role "class".

XPathSelector objects

A :class:`XPathSelector` object is a wrapper over response to select certain parts of its content.

System Message: ERROR/3 (<stdin>, line 20); backlink

Unknown interpreted text role "class".

A :class:`Request` object represents an HTTP request, which is usually generated in the Spider and executed by the Downloader, and thus generating a :class:`Response`.

System Message: ERROR/3 (<stdin>, line 23); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 23); backlink

Unknown interpreted text role "class".
url is a :class:`~scrapy.http.Response` object that will be used for

System Message: ERROR/3 (<stdin>, line 27); backlink

Unknown interpreted text role "class".

selecting and extracting data

XPathSelector Methods

System Message: ERROR/3 (<stdin>, line 34)

Unknown directive type "method".

.. method:: XPathSelector.x(xpath)

    Apply the given XPath relative to this XPathSelector and return a list
    of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList`) with
    the result.

    ``xpath`` is a string containing the XPath to apply

System Message: ERROR/3 (<stdin>, line 42)

Unknown directive type "method".

.. method:: XPathSelector.re(regex)

    Apply the given regex and return a list of unicode strings with the
    matches.

    ``regex`` can be either a compiled regular expression or a string which
    will be compiled to a regular expression using ``re.compile(regex)``

System Message: ERROR/3 (<stdin>, line 50)

Unknown directive type "method".

.. method:: XPathSelector.extract()

    Return a unicode string with the content of this :class:`XPathSelector`
    object.

System Message: ERROR/3 (<stdin>, line 55)

Unknown directive type "method".

.. method:: XPathSelector.extract_unquoted()

    Return a unicode string with the content of this :class:`XPathSelector`
    without entities or CDATA. This method is intended to be use for text-only
    selectors, like ``//h1/text()`` (but not ``//h1``). If it's used for
    :class:`XPathSelector` objects which don't select a textual content (ie. if
    they contain tags), the output of this method is undefined.

System Message: ERROR/3 (<stdin>, line 63)

Unknown directive type "method".

.. method:: XPathSelector.register_namespace(prefix, uri)

    Register the given namespace to be used in this :class:`XPathSelector`.
    Without registering namespaces you can't select or extract data from
    non-standard namespaces. See examples below.

System Message: ERROR/3 (<stdin>, line 69)

Unknown directive type "method".

.. method:: XPathSelector.__nonzero__()

    Returns ``True`` if there is any real content selected by this
    :class:`XPathSelector` or ``False`` otherwise.  In other words, the boolean
    value of an XPathSelector is given by the contents it selects.

XPathSelectorList objects

The :class:`XPathSelectorList` class is subclass of the builtin list class, which provides a few additional methods.

System Message: ERROR/3 (<stdin>, line 80); backlink

Unknown interpreted text role "class".

XPathSelectorList Methods

System Message: ERROR/3 (<stdin>, line 87)

Unknown directive type "method".

.. method:: XPathSelectorList.x(xpath)

    Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector`
    objects in this list and return their results flattened, as new
    :class:`XPathSelectorList`.

    ``xpath`` is the same argument as the one in :meth:`XPathSelector.x`

System Message: ERROR/3 (<stdin>, line 95)

Unknown directive type "method".

.. method:: XPathSelector.re(regex)

    Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector`
    objects in this list and return their results flattened, as a list of
    unicode strings.

    ``regex`` is the same argument as the one in :meth:`XPathSelector.re`

System Message: ERROR/3 (<stdin>, line 103)

Unknown directive type "method".

.. method:: XPathSelector.extract()

    Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector`
    objects in this list and return their results flattened, as a list of
    unicode strings.

System Message: ERROR/3 (<stdin>, line 109)

Unknown directive type "method".

.. method:: XPathSelector.extract_unquoted()

    Call the :meth:`XPathSelector.extract_unoquoted` method for all
    :class:`XPathSelector` objects in this list and return their results
    flattened, as a list of unicode strings. This method should not be applied
    to all kinds of XPathSelectors. For more info see
    :meth:`XPathSelector.extract_unoquoted`.

HtmlXPathSelector objects

A subclass of :class:`XPathSelector` for working with HTML content. It uses the libxml2 HTML parser. See the :class:`XPathSelector` API for more info.

System Message: ERROR/3 (<stdin>, line 122); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 122); backlink

Unknown interpreted text role "class".

HtmlXPathSelector examples

Here's a couple of :class:`HtmlXPathSelector` examples to illustrate several concepts. In all cases we assume there is already a :class:`HtmlPathSelector` instanced with a :class:`~scrapy.http.Response` object like this:

System Message: ERROR/3 (<stdin>, line 130); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 130); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 130); backlink

Unknown interpreted text role "class".
x = HtmlXPathSelector(html_response)

  1. Select all <h1> elements from a HTML response body, returning a list of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):

    System Message: ERROR/3 (<stdin>, line 136); backlink

    Unknown interpreted text role "class".

    System Message: ERROR/3 (<stdin>, line 136); backlink

    Unknown interpreted text role "class".

    x.x("//h1")

  2. Extract the text of all <h1> elements from a HTML response body, returning a list of unicode strings:

    x.x("//h1").extract()         # this includes the h1 tag
x.x("//h1/text()").extract()  # this excludes the h1 tag

  3. Iterate over all <p> tags and print their class attribute:

    for node in x.x("//p"):
...    print node.x("@href")

  4. Extract textual data from all <p> tags without entities, as a list of unicode strings:

    x.x("//p/text()").extract_unquoted()

# the following line is wrong. extract_unquoted() should only be used
# with textual XPathSelectors
x.x("//p").extract_unquoted()  # it may work but output is unpredictable

XmlXPathSelector objects

A subclass of :class:`XPathSelector` for working with XML content. It uses the libxml2 XML parser. See the :class:`XPathSelector` API for more info.

System Message: ERROR/3 (<stdin>, line 166); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 166); backlink

Unknown interpreted text role "class".

XmlXPathSelector examples

Here's a couple of :class:`XmlXPathSelector` examples to illustrate several concepts. In all cases we assume there is already a :class:`XmlPathSelector` instanced with a :class:`~scrapy.http.Response` object like this:

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 172); backlink

Unknown interpreted text role "class".
x = HtmlXPathSelector(xml_response)

  1. Select all <product> elements from a XML response body, returning a list of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):

    System Message: ERROR/3 (<stdin>, line 178); backlink

    Unknown interpreted text role "class".

    System Message: ERROR/3 (<stdin>, line 178); backlink

    Unknown interpreted text role "class".

    x.x("//h1")

  2. Extract all prices from a Google Base XML feed which requires registering a namespace:

    x.register_namespace("g", "http://base.google.com/ns/1.0")
x.x("//g:price").extract()
</html>