22 KiB
Selectors
When you're scraping web pages, the most common task you need to perform is to extract data from the HTML source. There are several libraries available to achieve this:
- BeautifulSoup is a very popular screen scraping library among Python programmers which constructs a Python object based on the structure of the HTML code and also deals with bad markup reasonably well, but it has one drawback: it's slow.
- lxml is a XML parsing library (which also parses HTML) with a pythonic API based on ElementTree (which is not part of the Python standard library).
Scrapy comes with its own mechanism for extracting data. They're called selectors because they "select" certain parts of the HTML document specified either by XPath or CSS expressions.
XPath is a language for selecting nodes in XML documents, which can also be used with HTML. CSS is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements.
Both lxml and Scrapy Selectors are built over the libxml2 library, which means they're very similar in speed and parsing accuracy.
This page explains how selectors work and describes their API which is very small and simple, unlike the lxml API which is much bigger because the lxml library can be used for many other tasks, besides selecting markup documents.
For a complete reference of the selectors API see :ref:`XPath selector reference <topics-xpath-selectors-ref>` and :ref:`CSS selector reference <topics-css-selectors-ref>`.
System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "ref".Using selectors
Constructing selectors
There are four types of selectors bundled with Scrapy. Those are:
:class:`~scrapy.selector.HtmlXPathSelector` - for working with HTML documents using XPath.
System Message: ERROR/3 (<stdin>, line 55); backlink
Unknown interpreted text role "class".
:class:`~scrapy.selector.XmlXPathSelector` - for working with XML documents using XPath.
System Message: ERROR/3 (<stdin>, line 58); backlink
Unknown interpreted text role "class".
:class:`~scrapy.selector.HtmlCSSSelector` - for working with HTML documents using CSS selectors.
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "class".
:class:`~scrapy.selector.XmlCSSSelector` - for working with XML documents using CSS selectors.
System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 67)
Unknown directive type "highlight".
.. highlight:: python
All of them share the same selector API, and are constructed with a Response object as their first parameter. This is the Response they're going to be "selecting".
Example:
hcs = HtmlCSSSelector(response) # an HTML CSS selector xxs = XmlXPathSelector(response) # an XML XPath selector
Using selectors
To explain how to use the selectors we'll use the Scrapy shell (which provides interactive testing) and an example page located in the Scrapy documentation server:
http://doc.scrapy.org/en/latest/_static/selectors-sample1.html
Here's its HTML code:
System Message: ERROR/3 (<stdin>, line 91)
Unknown directive type "literalinclude".
.. literalinclude:: ../_static/selectors-sample1.html :language: html
System Message: ERROR/3 (<stdin>, line 94)
Unknown directive type "highlight".
.. highlight:: sh
First, let's open the shell:
scrapy shell http://doc.scrapy.org/en/latest/_static/selectors-sample1.html
Then, after the shell loads, you'll have some selectors already instantiated and ready to use.
Since we're dealing with HTML, we can use either the :class:`~scrapy.selector.HtmlXPathSelector` object which is found, by default, in the hxs shell variable, or the equivalent :class:`~scrapy.selector.HtmlCSSSelector` found in the hcs shell variable. Note that CSS selectors can only select element nodes, while XPath selectors can select any nodes, including text and comment nodes. There are some methods to augment CSS selectors with XPath as we'll see below.
System Message: ERROR/3 (<stdin>, line 103); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 103); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 111)
Unknown directive type "highlight".
.. highlight:: python
So, by looking at the :ref:`HTML code <topics-selectors-htmlcode>` of that page, let's construct an XPath (using an HTML selector) for selecting the text inside the title tag:
System Message: ERROR/3 (<stdin>, line 113); backlink
Unknown interpreted text role "ref".>>> hxs.select('//title/text()')
[<HtmlXPathSelector (text) xpath=//title/text()>]
As you can see, the select() method returns an XPathSelectorList, which is a list of new selectors. This API can be used quickly for extracting nested data.
To actually extract the textual data, you must call the selector extract() method, as follows:
>>> hxs.select('//title/text()').extract()
[u'Example website']
Now notice that CSS selectors can't select the text nodes. There are some methods that allow enhancing CSS selectors, such as text and get:
>>> hcs.select('title').text()
[<HtmlCSSSelector xpath='text()' data=u'Example website'>]
>>> hcs.select('title').text().extract()
[u'Example website']
Now we're going to get the base URL and some image links:
>>> hxs.select('//base/@href').extract()
[u'http://example.com/']
>>> hcs.select('base').get('href')
[u'http://example.com/']
>>> hxs.select('//a[contains(@href, "image")]/@href').extract()
[u'image1.html',
u'image2.html',
u'image3.html',
u'image4.html',
u'image5.html']
>>> hcs.select('a[href*=image]').get('href').extract()
[u'image1.html',
u'image2.html',
u'image3.html',
u'image4.html',
u'image5.html']
>>> hxs.select('//a[contains(@href, "image")]/img/@src').extract()
[u'image1_thumb.jpg',
u'image2_thumb.jpg',
u'image3_thumb.jpg',
u'image4_thumb.jpg',
u'image5_thumb.jpg']
>>> hcs.select('a[href*=image] img').get('src').extract()
[u'image1_thumb.jpg',
u'image2_thumb.jpg',
u'image3_thumb.jpg',
u'image4_thumb.jpg',
u'image5_thumb.jpg']
Nesting selectors
The select() selector method returns a list of selectors of the same type (XPath or CSS), so you can call the select() for those selectors too. Here's an example:
>>> links = hxs.select('//a[contains(@href, "image")]')
>>> links.extract()
[u'<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
u'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
u'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
u'<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
u'<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
>>> for index, link in enumerate(links):
args = (index, link.select('@href').extract(), link.select('img/@src').extract())
print 'Link number %d points to url %s and image %s' % args
Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg']
Link number 1 points to url [u'image2.html'] and image [u'image2_thumb.jpg']
Link number 2 points to url [u'image3.html'] and image [u'image3_thumb.jpg']
Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg']
Link number 4 points to url [u'image5.html'] and image [u'image5_thumb.jpg']
The CSSSelectorList select method will accept CSS selectors, as expected, but it also provides an xpath method that accepts XPath selectors to augment the CSS selectors. Here's an example:
>>> links = hcs.select('a[href*=image]')
>>> for index, link in enumerate(links):
args = (index, link.get('href').extract(), link.xpath('img/@src').extract())
print 'Link number %d points to url %s and image %s' % args
Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg']
Link number 1 points to url [u'image2.html'] and image [u'image2_thumb.jpg']
Link number 2 points to url [u'image3.html'] and image [u'image3_thumb.jpg']
Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg']
Link number 4 points to url [u'image5.html'] and image [u'image5_thumb.jpg']
Using selectors with regular expressions
Selectors (both CSS and XPath) also have a re() method for extracting data using regular expressions. However, unlike using the select() method, the re() method does not return a list of :class:`~scrapy.selector.XPathSelector` objects, so you can't construct nested .re() calls.
System Message: ERROR/3 (<stdin>, line 218); backlink
Unknown interpreted text role "class".Here's an example used to extract images names from the :ref:`HTML code <topics-selectors-htmlcode>` above:
System Message: ERROR/3 (<stdin>, line 224); backlink
Unknown interpreted text role "ref".>>> hxs.select('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
[u'My image 1',
u'My image 2',
u'My image 3',
u'My image 4',
u'My image 5']
Working with relative XPaths
Keep in mind that if you are nesting XPathSelectors and use an XPath that starts with /, that XPath will be absolute to the document and not relative to the XPathSelector you're calling it from.
For example, suppose you want to extract all <p> elements inside <div> elements. First, you would get all <div> elements:
>>> divs = hxs.select('//div')
At first, you may be tempted to use the following approach, which is wrong, as it actually extracts all <p> elements from the document, not only those inside <div> elements:
>>> for p in divs.select('//p') # this is wrong - gets all <p> from the whole document
>>> print p.extract()
This is the proper way to do it (note the dot prefixing the .//p XPath):
>>> for p in divs.select('.//p') # extracts all <p> inside
>>> print p.extract()
Another common case would be to extract all direct <p> children:
>>> for p in divs.select('p')
>>> print p.extract()
For more details about relative XPaths see the Location Paths section in the XPath specification.
Built-in Selectors reference
System Message: ERROR/3 (<stdin>, line 276)
Unknown directive type "module".
.. module:: scrapy.selector :synopsis: Selectors classes
There are four types of selectors bundled with Scrapy: :class:`HtmlXPathSelector` and :class:`XmlXPathSelector`, :class:`HtmlCSSSelector` and :class:`XmlCSSSelector`. All of them implement the same :class:`XPathSelector` interface. The only differences are the selector syntax and whether it is used to process HTML data or XML data.
System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".XPathSelector objects
A :class:`XPathSelector` object is a wrapper over response to select certain parts of its content.
System Message: ERROR/3 (<stdin>, line 292); backlink
Unknown interpreted text role "class".response is a :class:`~scrapy.http.Response` object that will be used for selecting and extracting data.
System Message: ERROR/3 (<stdin>, line 295); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 298)
Unknown directive type "method".
.. method:: select(xpath)
Apply the given XPath relative to this XPathSelector and return a list
of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList`)
with the result.
``xpath`` is a string containing the XPath to apply
System Message: ERROR/3 (<stdin>, line 306)
Unknown directive type "method".
.. method:: re(regex)
Apply the given regex and return a list of unicode strings with the
matches.
``regex`` can be either a compiled regular expression or a string which
will be compiled to a regular expression using ``re.compile(regex)``
System Message: ERROR/3 (<stdin>, line 314)
Unknown directive type "method".
.. method:: extract()
Return a unicode string with the content of this :class:`XPathSelector`
object.
System Message: ERROR/3 (<stdin>, line 319)
Unknown directive type "method".
.. method:: register_namespace(prefix, uri)
Register the given namespace to be used in this :class:`XPathSelector`.
Without registering namespaces you can't select or extract data from
non-standard namespaces. See examples below.
System Message: ERROR/3 (<stdin>, line 325)
Unknown directive type "method".
.. method:: remove_namespaces()
Remove all namespaces, allowing to traverse the document using
namespace-less xpaths. See example below.
System Message: ERROR/3 (<stdin>, line 330)
Unknown directive type "method".
.. method:: __nonzero__()
Returns ``True`` if there is any real content selected by this
:class:`XPathSelector` or ``False`` otherwise. In other words, the
boolean value of an XPathSelector is given by the contents it selects.
XPathSelectorList objects
The :class:`XPathSelectorList` class is subclass of the builtin list class, which provides a few additional methods.
System Message: ERROR/3 (<stdin>, line 341); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 344)
Unknown directive type "method".
.. method:: select(xpath)
Call the :meth:`XPathSelector.select` method for all
:class:`XPathSelector` objects in this list and return their results
flattened, as a new :class:`XPathSelectorList`.
``xpath`` is the same argument as the one in
:meth:`XPathSelector.select`
System Message: ERROR/3 (<stdin>, line 353)
Unknown directive type "method".
.. method:: re(regex)
Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector`
objects in this list and return their results flattened, as a list of
unicode strings.
``regex`` is the same argument as the one in :meth:`XPathSelector.re`
System Message: ERROR/3 (<stdin>, line 361)
Unknown directive type "method".
.. method:: extract()
Call the :meth:`XPathSelector.extract` method for all
:class:`XPathSelector` objects in this list and return their results
flattened, as a list of unicode strings.
System Message: ERROR/3 (<stdin>, line 367)
Unknown directive type "method".
.. method:: extract_unquoted()
Call the :meth:`XPathSelector.extract_unoquoted` method for all
:class:`XPathSelector` objects in this list and return their results
flattened, as a list of unicode strings. This method should not be
applied to all kinds of XPathSelectors. For more info see
:meth:`XPathSelector.extract_unoquoted`.
HtmlXPathSelector objects
A subclass of :class:`XPathSelector` for working with HTML content. It uses the libxml2 HTML parser. See the :class:`XPathSelector` API for more info.
System Message: ERROR/3 (<stdin>, line 380); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 380); backlink
Unknown interpreted text role "class".HtmlXPathSelector examples
Here's a couple of :class:`HtmlXPathSelector` examples to illustrate several concepts. In all cases, we assume there is already an :class:`HtmlXPathSelector` instantiated with a :class:`~scrapy.http.Response` object like this:
System Message: ERROR/3 (<stdin>, line 389); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 389); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 389); backlink
Unknown interpreted text role "class".x = HtmlXPathSelector(html_response)
Select all <h1> elements from a HTML response body, returning a list of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):
System Message: ERROR/3 (<stdin>, line 396); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 396); backlink
Unknown interpreted text role "class".
x.select("//h1")Extract the text of all <h1> elements from a HTML response body, returning a list of unicode strings:
x.select("//h1").extract() # this includes the h1 tag x.select("//h1/text()").extract() # this excludes the h1 tagIterate over all <p> tags and print their class attribute:
for node in x.select("//p"): ... print node.select("@class").extract()Extract textual data from all <p> tags without entities, as a list of unicode strings:
x.select("//p/text()").extract_unquoted() # the following line is wrong. extract_unquoted() should only be used # with textual XPathSelectors x.select("//p").extract_unquoted() # it may work but output is unpredictable
XmlXPathSelector objects
A subclass of :class:`XPathSelector` for working with XML content. It uses the libxml2 XML parser. See the :class:`XPathSelector` API for more info.
System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "class".XmlXPathSelector examples
Here's a couple of :class:`XmlXPathSelector` examples to illustrate several concepts. In both cases we assume there is already an :class:`XmlXPathSelector` instantiated with a :class:`~scrapy.http.Response` object like this:
System Message: ERROR/3 (<stdin>, line 432); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 432); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 432); backlink
Unknown interpreted text role "class".x = XmlXPathSelector(xml_response)
Select all <product> elements from a XML response body, returning a list of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):
System Message: ERROR/3 (<stdin>, line 438); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 438); backlink
Unknown interpreted text role "class".
x.select("//product")Extract all prices from a Google Base XML feed which requires registering a namespace:
x.register_namespace("g", "http://base.google.com/ns/1.0") x.select("//g:price").extract()
Removing namespaces
When dealing with scraping projects, it is often quite convenient to get rid of namespaces altogether and just work with element names, to write more simple/convenient XPaths. You can use the :meth:`XPathSelector.remove_namespaces` method for that.
System Message: ERROR/3 (<stdin>, line 455); backlink
Unknown interpreted text role "meth".Let's show an example that illustrates this with Github blog atom feed.
First, we open the shell with the url we want to scrape:
$ scrapy shell https://github.com/blog.atom
Once in the shell we can try selecting all <link> objects and see that it doesn't work (because the Atom XML namespace is obfuscating those nodes):
>>> xxs.select("//link")
[]
But once we call the :meth:`XPathSelector.remove_namespaces` method, all nodes can be accessed directly by their names:
System Message: ERROR/3 (<stdin>, line 472); backlink
Unknown interpreted text role "meth".>>> xxs.remove_namespaces()
>>> xxs.select("//link")
[<XmlXPathSelector xpath='//link' data=u'<link xmlns="http://www.w3.org/2005/Atom'>,
<XmlXPathSelector xpath='//link' data=u'<link xmlns="http://www.w3.org/2005/Atom'>,
...
If you wonder why the namespace removal procedure is not always called, instead of having to call it manually. This is because of two reasons which, in order of relevance, are:
- removing namespaces requires to iterate and modify all nodes in the document, which is a reasonably expensive operation to performs for all documents crawled by Scrapy
- there could be some cases where using namespaces is actually required, in case some element names clash between namespaces. These cases are very rare though.
CSSSelector objects
A :class:`CSSSelectorMixin` object is a mixin for either :class:`XmlXPathSelector` or :class:`HtmlXPathSelector` to select element nodes using CSS selectors syntax. As a mixin, it is not meant to be used on its own, but as a secondary parent class. See :class:`XmlCSSSelector` and :class:`HtmlCSSSelector` for implementations.
System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 508)
Unknown directive type "method".
.. method:: select(css)
Apply the given CSS selector relative to this CSSSelectorMixin and
return a list of :class:`CSSSelectorMixin` objects (ie. a
:class:`CSSSelectorList`) with the result.
``css`` is a string containing the CSS selector to apply.
System Message: ERROR/3 (<stdin>, line 516)
Unknown directive type "method".
.. method:: xpath(xpath)
Apply the given XPath relative to this CSSSelectorMixin and return a list
of :class:`CSSSelectorMixin` objects (ie. a :class:`CSSSelectorList`)
with the result.
``xpath`` is a string containing the XPath to apply.
System Message: ERROR/3 (<stdin>, line 524)
Unknown directive type "method".
.. method:: get(attr)
Get the attribute relative to this CSSSelectorMixin and return a list
of :class:`CSSSelectorMixin` objects (ie. a :class:`CSSSelectorList`)
with the result (usually with one element only).
``attr`` is a string containing the attribute name to get.
System Message: ERROR/3 (<stdin>, line 532)
Unknown directive type "method".
.. method:: text(all=False)
Get the children text nodes relative to this CSSSelectorMixin or, if
``all`` is True, a string node concatenating all of the descendant text
nodes relative to this CSSSelectorMixin, and return a list of
:class:`CSSSelectorMixin` objects (ie. a :class:`CSSSelectorList`) with
the result.
``all`` is a boolean to either select children text nodes (False) or
select a string node concatenating all of the descendant text nodes.
CSSSelectorList objects
The :class:`CSSSelectorList` class is subclass of :class:`XPathSelectorList` which overrides and adds methods to match those of :class:`CSSSelectorMixin`.
System Message: ERROR/3 (<stdin>, line 548); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 548); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 548); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 552)
Unknown directive type "method".
.. method:: xpath(xpath)
Call the :meth:`CSSSelectorMixin.xpath` method for all
:class:`CSSSelectorMixin` objects in this list and return their results
flattened, as a new :class:`CSSSelectorList`.
``xpath`` is the same argument as the one in
:meth:`CSSSelectorMixin.xpath`
System Message: ERROR/3 (<stdin>, line 561)
Unknown directive type "method".
.. method:: get(attr)
Call the :meth:`CSSSelectorMixin.get` method for all
:class:`CSSSelectorMixin` objects in this list and return their results
flattened, as a new :class:`CSSSelectorList`.
``attr`` is the same argument as the one in :meth:`CSSSelectorMixin.get`
System Message: ERROR/3 (<stdin>, line 569)
Unknown directive type "method".
.. method:: text(all=False)
Call the :meth:`CSSSelectorMixin.text` method for all
:class:`CSSSelectorMixin` objects in this list and return their results
flattened, as a new :class:`CSSSelectorList`.
``all`` is the same argument as the one in :meth:`CSSSelectorMixin.text`
HtmlCSSSelector objects
A subclass of :class:`CSSSelectorMixin` and :class:`HtmlXPathSelector` for working with HTML content using CSS selectors.
System Message: ERROR/3 (<stdin>, line 582); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 582); backlink
Unknown interpreted text role "class".HtmlCSSSelector examples
Here's a couple of :class:`HtmlCSSSelector` examples to illustrate several concepts. In all cases, we assume there is already an :class:`HtmlCSSSelector` instantiated with a :class:`~scrapy.http.Response` object like this:
System Message: ERROR/3 (<stdin>, line 588); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 588); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 588); backlink
Unknown interpreted text role "class".x = HtmlCSSSelector(html_response)
Select all <h1> elements from a HTML response body, returning a list of :class:`HtmlCSSSelector` objects (ie. a :class:`CSSSelectorList` object):
System Message: ERROR/3 (<stdin>, line 594); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 594); backlink
Unknown interpreted text role "class".
x.select("h1")Extract the text of all <h1> elements from a HTML response body, returning a list of unicode strings:
x.select("h1").extract() # this includes the h1 tag x.select("h1").text().extract() # this excludes the h1 tagIterate over all <p> tags and print their class attribute:
for node in x.select("p"): ... print node.get("class").extract()
XmlCSSSelector objects
A subclass of :class:`CSSSelectorMixin` and :class:`XmlXPathSelector` for working with XML content using CSS selectors.
System Message: ERROR/3 (<stdin>, line 615); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 615); backlink
Unknown interpreted text role "class".XmlCSSSelector examples
Here's a couple of :class:`XmlCSSSelector` examples to illustrate several concepts. In both cases we assume there is already an :class:`XmlCSSSelector` instantiated with a :class:`~scrapy.http.Response` object like this:
System Message: ERROR/3 (<stdin>, line 621); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 621); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 621); backlink
Unknown interpreted text role "class".x = XmlCSSSelector(xml_response)
Select all <product> elements from a XML response body, returning a list of :class:`XmlCSSSelector` objects (ie. a :class:`CSSSelectorList` object):
System Message: ERROR/3 (<stdin>, line 627); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 627); backlink
Unknown interpreted text role "class".
x.select("product")Extract all prices from a Google Base XML feed which requires registering a namespace:
x.register_namespace("g", "http://base.google.com/ns/1.0") x.xpath("//g:price").extract()