diff --git a/scrapy/trunk/docs/ref/index.rst b/scrapy/trunk/docs/ref/index.rst index 880231f65..0a5746f05 100644 --- a/scrapy/trunk/docs/ref/index.rst +++ b/scrapy/trunk/docs/ref/index.rst @@ -9,6 +9,7 @@ This section documents the API of Scrapy |version|. For more information see :re :maxdepth: 1 spiders + selectors exceptions request-response extension-manager diff --git a/scrapy/trunk/docs/ref/request-response.rst b/scrapy/trunk/docs/ref/request-response.rst index 591742f9c..bbb4ec9af 100644 --- a/scrapy/trunk/docs/ref/request-response.rst +++ b/scrapy/trunk/docs/ref/request-response.rst @@ -75,8 +75,10 @@ Request objects be filtered by the scheduler. This is used when you want to perform an identical request multiple times, for whatever reason - ``errback`` is a function that will be called if any exception was raised while - processing the request, it takes a `Twisted Failure`_ instance as first parameter. + ``errback`` is a function that will be called if any exception was raised + while processing the request in Scrapy. This includes pages that failed + with 404 HTTP errors and such. , it receives a `Twisted Failure`_ + instance as first parameter. .. _Twisted Failure: http://twistedmatrix.com/documents/8.2.0/api/twisted.python.failure.Failure.html diff --git a/scrapy/trunk/docs/ref/selectors.rst b/scrapy/trunk/docs/ref/selectors.rst new file mode 100644 index 000000000..0ee82f687 --- /dev/null +++ b/scrapy/trunk/docs/ref/selectors.rst @@ -0,0 +1,186 @@ +.. _ref-selectors: + +============= +Selectors API +============= + +There are two types of selectors bundled with Scrapy: +:class:`HtmlXPathSelector` and :class:`XmlXPathSelector`. Both of them +implement the same :class:`XPathSelector` interface. The only different is that +one is used to process HTML data and the other XML data. + +XPathSelector objects +===================== + +.. class:: XPathSelector(response) + + A :class:`XPathSelector` object is a wrapper over response to select + certain parts of its content. + + A :class:`Request` object represents an HTTP request, which is usually + generated in the Spider and executed by the Downloader, and thus generating + a :class:`Response`. + + ``url`` is a :class:`~scrapy.http.Response` object that will be used for + selecting and extracting data + + +XPathSelector Methods +--------------------- + +.. method:: XPathSelector.x(xpath) + + Apply the given XPath relative to this XPathSelector and return a list + of :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList`) with + the result. + + ``xpath`` is a string containing the XPath to apply + +.. method:: XPathSelector.re(regex) + + Apply the given regex and return a list of unicode strings with the + matches. + + ``regex`` can be either a compiled regular expression or a string which + will be compiled to a regular expression using ``re.compile(regex)`` + +.. method:: XPathSelector.extract() + + Return a unicode string with the content of this :class:`XPathSelector` + object. + +.. method:: XPathSelector.extract_unquoted() + + Return a unicode string with the content of this :class:`XPathSelector` + without entities or CDATA. This method is intended to be use for text-only + selectors, like ``//h1/text()`` (but not ``//h1``). If it's used for + :class:`XPathSelector` objects which don't select a textual content (ie. if + they contain tags), the output of this method is undefined. + +.. method:: XPathSelector.register_namespace(prefix, uri) + + Register the given namespace to be used in this :class:`XPathSelector`. + Without registering namespaces you can't select or extract data from + non-standard namespaces. See examples below. + +.. method:: XPathSelector.__nonzero__() + + Returns ``True`` if there is any real content selected by this + :class:`XPathSelector` or ``False`` otherwise. In other words, the boolean + value of an XPathSelector is given by the contents it selects. + +XPathSelectorList objects +========================= + +.. class:: XPathSelectorList + + The :class:`XPathSelectorList` class is subclass of the builtin ``list`` + class, which provides a few additional methods. + + +XPathSelectorList Methods +------------------------- + +.. method:: XPathSelectorList.x(xpath) + + Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` + objects in this list and return their results flattened, as new + :class:`XPathSelectorList`. + + ``xpath`` is the same argument as the one in :meth:`XPathSelector.x` + +.. method:: XPathSelector.re(regex) + + Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` + objects in this list and return their results flattened, as a list of + unicode strings. + + ``regex`` is the same argument as the one in :meth:`XPathSelector.re` + +.. method:: XPathSelector.extract() + + Call the :meth:`XPathSelector.re` method for all :class:`XPathSelector` + objects in this list and return their results flattened, as a list of + unicode strings. + +.. method:: XPathSelector.extract_unquoted() + + Call the :meth:`XPathSelector.extract_unoquoted` method for all + :class:`XPathSelector` objects in this list and return their results + flattened, as a list of unicode strings. This method should not be applied + to all kinds of XPathSelectors. For more info see + :meth:`XPathSelector.extract_unoquoted`. + +HtmlXPathSelector objects +========================= + +.. class:: HtmlXPathSelector(response) + + A subclass of :class:`XPathSelector` for working with HTML content. It uses + the `libxml2`_ HTML parser. See the :class:`XPathSelector` API for more info. + +.. _libxml2: http://xmlsoft.org/ + +HtmlXPathSelector examples +-------------------------- + +Here's a couple of :class:`HtmlXPathSelector` examples to illustrate several +concepts. In all cases we assume there is already a :class:`HtmlPathSelector` +instanced with a :class:`~scrapy.http.Response` object like this:: + + x = HtmlXPathSelector(html_response) + +1. Select all ``

`` elements from a HTML response body, returning a list of + :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):: + + x.x("//h1") + +2. Extract the text of all ``

`` elements from a HTML response body, + returning a list of unicode strings:: + + x.x("//h1").extract() # this includes the h1 tag + x.x("//h1/text()").extract() # this excludes the h1 tag + +3. Iterate over all ``

`` tags and print their class attribute:: + + for node in x.x("//p"): + ... print node.x("@href") + +4. Extract textual data from all ``

`` tags without entities, as a list of + unicode strings:: + + x.x("//p/text()").extract_unquoted() + + # the following line is wrong. extract_unquoted() should only be used + # with textual XPathSelectors + x.x("//p").extract_unquoted() # it may work but output is unpredictable + +XmlXPathSelector objects +======================== + +.. class:: XmlXPathSelector(response) + + A subclass of :class:`XPathSelector` for working with XML content. It uses + the `libxml2`_ XML parser. See the :class:`XPathSelector` API for more info. + +XmlXPathSelector examples +------------------------- + +Here's a couple of :class:`XmlXPathSelector` examples to illustrate several +concepts. In all cases we assume there is already a :class:`XmlPathSelector` +instanced with a :class:`~scrapy.http.Response` object like this:: + + x = HtmlXPathSelector(xml_response) + +1. Select all ```` elements from a XML response body, returning a list of + :class:`XPathSelector` objects (ie. a :class:`XPathSelectorList` object):: + + x.x("//h1") + +2. Extract all prices from a `Google Base XML feed`_ which requires registering + a namespace:: + + x.register_namespace("g", "http://base.google.com/ns/1.0") + x.x("//g:price").extract() + +.. _Google Base XML feed: http://base.google.com/support/bin/answer.py?hl=en&answer=59461 diff --git a/scrapy/trunk/docs/topics/selectors.rst b/scrapy/trunk/docs/topics/selectors.rst index 6358397ac..cf59d7200 100644 --- a/scrapy/trunk/docs/topics/selectors.rst +++ b/scrapy/trunk/docs/topics/selectors.rst @@ -4,65 +4,113 @@ Selectors ========= -Selectors are the recommended way to extract information from documents. They retrieve information from the response's body, given an XPath, or a Regular Expression that you provide. +Introduction +------------ + +When you're scraping web pages, the most common task you need to perform is +extract data from the HTML source. There are several libraries available to +achieve this: + + * `BeautifulSoup`_ is a very popular screen scraping library among Python + programmers which constructs a Python object based on the + structure of the HTML code and also deals with bad markup reasonable well, + but it has one drawback: it's slow. + + * `lxml`_ is a XML parsing library (which also parses HTML) with a pythonic + API based on `ElementTree`_ (which is not part of the Python standard + library). + +Scrapy comes with its own mechanism for mechanism for extracting data. They're +called selectors, because they "select" certain parts of the HTML document +specified by XPaths. + +`XPath`_ is a language for selecting nodes in XML documents, and it's also +applicable for HTML documents. + +Both `lxml`_ and Scrapy Selectors are built over the `libxml2`_ library, which +means they're very similar in speed and parsing accuracy. + +This page explains how selectors work and describes their API which is very +small and simple, unlike the `lxml`_ API which is much bigger because the +`lxml`_ library can be use for many other tasks, besides selecting markup +documents. + +For a complete reference of the selectors API see the :ref:`XPath selector +reference `. + +.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ +.. _lxml: http://codespeak.net/lxml/ +.. _ElementTree: http://docs.python.org/library/xml.etree.elementtree.html +.. _libxml2: http://xmlsoft.org/ +.. _XPath: http://www.w3.org/TR/xpath + +Constructing selectors +---------------------- + +There are two types of selectors bundled with Scrapy. Those are: + + * :class:`~scrapy.xpath.HtmlXPathSelector` - for working with HTML documents + + * :class:`~scrapy.xpath.XmlXPathSelector` - for working with XML documents .. highlight:: python -Currently there are two kinds of selectors, HtmlXPathSelectors, and XmlXPathSelectors. Both work in the same way; they are first instanciated with a response, for example:: +Both share the same selector API, and are constructed with a Response object as +its first parameter. This is the Response they're gonna be "selecting". - hxs = HtmlXPathSelector(response) # an HTML selector - xxs = XmlXPathSelector(response) # an XML selector +Example:: -.. highlight:: sh + hxs = HtmlXPathSelector(response) # a HTML selector + xxs = XmlXPathSelector(response) # a XML selector -Now, before going on with selectors, I'd suggest you to open a Scrapy shell, which you can use by calling your project manager with the 'shell' argument; something like:: +Using selectors with XPaths +--------------------------- - $ ./scrapy-ctl.py shell +To explain how to use the selectors we'll use the `Scrapy shell` (which +provides interactive testing) and an example page located in Scrapy +documentation server: -Notice that you'll have to install IPython in order to use this feature, but believe me that it worths it; the shell is **very** useful. + http://doc.scrapy.org/_static/selectors-sample1.html -With the shell you can simulate parsing a webpage, either by calling "scrapy-ctl.py shell" with an url as an additional parameter, or by using the shell's 'get' command, which tries -to retreive the given url, and fills in the 'response' variable with the result. +.. _topics-selectors-htmlcode: -Ok, so now let's use the shell to show you a bit how do selectors work. - -We'll use an example page located in Scrapy's (here's a `direct link <../_static/selectors-sample1.html>`_ if you want to download it), whose markup is: +Here's its HTML code: .. literalinclude:: ../_static/selectors-sample1.html :language: html .. highlight:: sh -First, we open the shell:: +First, let's open the shell:: - $ ./scrapy-ctl.py shell 'http://www.scrapy.org/docs/topics/sample1.htm' + scrapy-ctl.py shell http://doc.scrapy.org/_static/selectors-sample1.html -Then, after the shell loads, you'll have some already-made objects for you to play with. Two of them, hxs and xxs, are selectors. +Then, after the shell loads, you'll have some selectors already instanced and +ready to use. + +Since we're dealing with HTML we'll be using the +:class:`~scrapy.xpath.HtmlXPathSelector` object which is found, by default, in +the ``hxs`` shell variable. .. highlight:: python -You could instanciate your own by doing:: - - from scrapy.xpath.selector import HtmlXPathSelector, XmlXPathSelector - my_html_selector = HtmlXPathSelector(response) - my_xml_selector = XmlXPathSelector(response) - -Where 'response' is the object that Scrapy already created for you containing the given url's response. - -But anyway, we'll stick to the selectors that Scrapy already made for us, and more specifically, the HtmlXPathSelector (since we're working with an HTML document right now). - -Let's try extracting the title:: +So, by looking at the :ref:`HTML code ` of that page +let's construct an XPath (using an HTML selector) for selecting the text inside +the title tag:: >>> hxs.x('//title/text()') [] -As you can see, the x method returns an XPathSelectorList, which is actually a list of selectors. -To extract their data you must use the extract() method, as follows:: +As you can see, the x() method returns a XPathSelectorList, which is a list of +new selectors. This API can be used quickly for extracting nested data. + +To actually extract the textual data you must call the selector ``extract()`` +method, as follows:: >>> hxs.x('//title/text()').extract() [u'Example website'] -Let's know extract the base URL and some image links:: +Now we're going to get the base URL and some image links:: >>> hxs.x('//base/@href').extract() [u'http://example.com/'] @@ -81,8 +129,17 @@ Let's know extract the base URL and some image links:: u'image4_thumb.jpg', u'image5_thumb.jpg'] -And here's an example which shows the `re()` method of xpath selectors which -allows you to use regular expressions to select parts. + +Using selectors with regular expressions +---------------------------------------- + +Selectors also have a ``re()`` method for extracting data using regular +expressions. However, unlike using the ``x()`` method, the ``re()`` method does +not return a list of :class:`~scrapy.xpath.XPathSelector` objects, so you can't +construct nested ``.re()`` calls. + +Here's an example used to extract images names from the :ref:`HTML code +` above:: >>> hxs.x('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)') [u'My image 1', @@ -92,13 +149,11 @@ allows you to use regular expressions to select parts. u'My image 5'] -Now let's explain a bit what we just did. +Nesting selectors +----------------- -Selector's x() method, is intended to select a node or an attribute from the -document, given an XPath expression, as you could see upwards. - -You can apply an x() call to any node you have, which means that you can join -different calls, for example::: +The ``x()`` selector method returns a list of selectors, so you can call the +``x()`` for those selectors too. Here's an example:: >>> links = hxs.x('//a[contains(@href, "image")]') >>> links.extract() @@ -109,7 +164,8 @@ different calls, for example::: u'Name: My image 5
'] >>> for index, link in enumerate(links): - print 'Link number %d points to url %s and image %s' % (index, link.x('@href').extract(), link.x('img/@src').extract()) + args = (index, link.x('@href').extract(), link.x('img/@src').extract()) + print 'Link number %d points to url %s and image %s' % args Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg'] Link number 1 points to url [u'image2.html'] and image [u'image2_thumb.jpg'] @@ -117,16 +173,3 @@ different calls, for example::: Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg'] Link number 4 points to url [u'image5.html'] and image [u'image5_thumb.jpg'] -There are some things to keep in mind here: - -1. | x() calls always return an XPathSelectorList, which is basically a list of selectors, with the extra ability of applying XPath or Regexp to each of its items and - returning a new list. - | That's why you can concatenate x() calls, because they always return XPathSelectorLists, and you can always reapply that method over them. -2. x() calls are relative to the node your standing on, so selector.x('body/div[@id="mydiv"]') equals selector.x('body').x('div[@id="mydiv"]'). -3. The extract() method *always* returns a list, even if it contains only one element. Don't forget that. - -| You may also have noticed that I've used another method up there; the re() method. -| This one is very useful when the data extracted by XPath is not enough and you *have to* (remember to not abuse of regexp) make an extra parsing of the information you've got. -| In this cases, you just apply the re() method over any XPathSelector/XPathSelectorList you have with a compiled regexp pattern as the only argument, or a string with the pattern to be compiled. -| Remember that the re() method *always* returns an already extracted list, which means that you can't go back to a node from the result of a re() call. - diff --git a/scrapy/trunk/docs/topics/shell.rst b/scrapy/trunk/docs/topics/shell.rst index 2005234ef..c9f2d6476 100644 --- a/scrapy/trunk/docs/topics/shell.rst +++ b/scrapy/trunk/docs/topics/shell.rst @@ -61,9 +61,9 @@ These commands can be typed without the leading percent sign if you have Custom Shell Objects -------------------- -The console automatically makes some useful Scrapy objects available for the -downloaded page, like the Response object and the XPath selectors (for both -HTML and XML content). +The console automatically creates some useful Scrapy objects for the downloaded +page, like the :class:`~scrapy.http.Response` object and the +:class:`~scrapy.xpath.XPathSelector` objects (for both HTML and XML content). Those objects are: @@ -75,9 +75,11 @@ Those objects are: * ``response`` - a Response object of the downloaded page - * ``hxs`` - a HtmlXPathSelector object for the Response of the downloaded page + * ``hxs`` - a :class:`~scrapy.xpath.HtmlXPathSelector` object for the Response + of the downloaded page - * ``xxs`` - a XmlXPathSelector object for the Response of the downloaded page + * ``xxs`` - a :class:`~scrapy.xpath.XmlXPathSelector` object for the Response + of the downloaded page * ``get ``- download a new response from the given URL and update all Scrapy objects accordingly