From 7fdfdb7fa244ed44274aaa9750efec1c333eedd1 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 22 Aug 2018 17:34:44 +0500 Subject: [PATCH] DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction --- docs/topics/selectors.rst | 514 +++++++++++++++++++++----------------- 1 file changed, 281 insertions(+), 233 deletions(-) diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index cf1b0ba22..2db982b37 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -25,17 +25,14 @@ either by `XPath`_ or `CSS`_ expressions. used with HTML. `CSS`_ is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements. -Scrapy selectors are powered by `parsel`_ library, which uses `lxml`_ library -under the hood. It means Scrapy selectors are very similar in speed and -parsing accuracy to lxml. +.. note:: + Scrapy Selectors is a thin wrapper around `parsel`_ library; the purpose of + this wrapper is to provide better integration with Scrapy Response objects. -This page explains how selectors work and describes their API which is very -small and simple, unlike the `lxml`_ API which is much bigger because the -`lxml`_ library can be used for many other tasks, besides selecting markup -documents. - -For a complete reference of the selectors API see -:ref:`Selector reference ` + `parsel`_ is a stand-alone web scraping library which can be used without + Scrapy. It uses `lxml`_ library under the hood, and implements an + easy API on top of lxml API. It means Scrapy selectors are very similar + in speed and parsing accuracy to lxml. .. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/ .. _lxml: http://lxml.de/ @@ -73,9 +70,8 @@ Constructing from response:: >>> Selector(response=response).xpath('//span/text()').get() 'good' -For convenience, response objects expose a selector on `.selector` attribute, -it's totally OK to use this shortcut when possible. By using it you can -ensure the response body is parsed only once:: +For convenience, response objects expose a selector on `.selector` attribute. +By using it you can ensure the response body is parsed only once:: >>> response.selector.xpath('//span/text()').get() 'good' @@ -88,8 +84,10 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:: >>> response.css('span::text').get() 'good' -Usually there is no need to construct Scrapy selectors manually because of -these shortcuts. +Usually there is no need to construct Scrapy selectors manually: +``response`` object is available in Spider callbacks, so in most cases +it is more convenient to use ``response.css()`` and ``response.xpath()`` +shortcuts. Using selectors --------------- @@ -392,6 +390,71 @@ Use it to extract just the first matching string:: >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)') 'My image 1' +.. _old-extraction-api: + +extract() and extract_first() +----------------------------- + +If you're a long-time Scrapy user, you're probably familiar +with ``.extract()`` and ``.extract_first()`` selector methods. Many blog posts +and tutorials are using them as well. These methods are still supported +by Scrapy, there are **no plans** to deprecate them. + +However, Scrapy usage docs are now written using ``.get()`` and +``.getall()`` methods. We feel that these new methods result in a more concise +and readable code. + +The following examples show how these methods map to each other. + +1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``:: + + >>> response.css('a::attr(href)').get() + 'image1.html' + >>> response.css('a::attr(href)').extract_first() + 'image1.html' + +2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``:: + + >>> response.css('a::attr(href)').getall() + ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] + >>> response.css('a::attr(href)').extract() + ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] + +2. ``Selector.get()`` is the same as ``Selector.extract()``:: + + >>> response.css('a::attr(href)')[0].get() + 'image1.html' + >>> response.css('a::attr(href)')[0].extract() + 'image1.html' + +4. For consistency, there is also ``Selector.getall()``, which returns a list:: + + >>> response.css('a::attr(href)')[0].getall() + ['image1.html'] + +So, the main difference is that output of ``.get()`` and ``.getall()`` methods +is more predictable: ``.get()`` always returns a single result, ``.getall()`` +always returns a list of all extracted results. With ``.extract()`` method +it was not always obvious if a result is a list or not; to get a single +result either ``.extract()`` or ``.extract_first()`` should be called. + + +.. _topics-selectors-xpaths: + +Working with XPaths +=================== + +Here are some tips which may help you to use XPath with Scrapy selectors +effectively. If you are not much familiar with XPath yet, +you may want to take a look first at this `XPath tutorial`_. + +.. note:: + Some of the tips are based on `this post from ScrapingHub's blog`_. + +.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html +.. _`this post from ScrapingHub's blog`: https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/ + + .. _topics-selectors-relative-xpaths: Working with relative XPaths @@ -428,6 +491,114 @@ XPath specification. .. _Location Paths: https://www.w3.org/TR/xpath#location-paths +When querying by class, consider using CSS +------------------------------------------ + +Because an element can contain multiple CSS classes, the XPath way to select elements +by class is the rather verbose:: + + *[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')] + +If you use ``@class='someclass'`` you may end up missing elements that have +other classes, and if you just use ``contains(@class, 'someclass')`` to make up +for that you may end up with more elements that you want, if they have a different +class name that shares the string ``someclass``. + +As it turns out, Scrapy selectors allow you to chain selectors, so most of the time +you can just select by class using CSS and then switch to XPath when needed:: + + >>> from scrapy import Selector + >>> sel = Selector(text='
') + >>> sel.css('.shout').xpath('./time/@datetime').getall() + ['2014-07-23 19:00'] + +This is cleaner than using the verbose XPath trick shown above. Just remember +to use the ``.`` in the XPath expressions that will follow. + +Beware of the difference between //node[1] and (//node)[1] +---------------------------------------------------------- + +``//node[1]`` selects all the nodes occurring first under their respective parents. + +``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text=""" + ....: + ....: """) + >>> xp = lambda x: sel.xpath(x).getall() + +This gets all first ``
  • `` elements under whatever it is its parent:: + + >>> xp("//li[1]") + ['
  • 1
    • ', '
  • 4
    • '] + +And this gets the first ``
  • `` element in the whole document:: + + >>> xp("(//li)[1]") + ['
  • 1
    • '] + +This gets all first ``
  • `` elements under an ``
      `` parent:: + + >>> xp("//ul/li[1]") + ['
    • 1
      • ', '
    • 4
      • '] + +And this gets the first ``
    • `` element under an ``
        `` parent in the whole document:: + + >>> xp("(//ul/li)[1]") + ['
      • 1
        • '] + +Using text nodes in a condition +------------------------------- + +When you need to use the text content as argument to an `XPath string function`_, +avoid using ``.//text()`` and use just ``.`` instead. + +This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*. +And when a node-set is converted to a string, which happens when it is passed as argument to +a string function like ``contains()`` or ``starts-with()``, it results in the text for the first element only. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text='Click here to go to the Next Page') + +Converting a *node-set* to string:: + + >>> sel.xpath('//a//text()').getall() # take a peek at the node-set + ['Click here to go to the ', 'Next Page'] + >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string + ['Click here to go to the '] + +A *node* converted to a string, however, puts together the text of itself plus of all its descendants:: + + >>> sel.xpath("//a[1]").getall() # select the first node + ['Click here to go to the Next Page'] + >>> sel.xpath("string(//a[1])").getall() # convert it to string + ['Click here to go to the Next Page'] + +So, using the ``.//text()`` node-set won't select anything in this case:: + + >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall() + [] + +But using the ``.`` to mean the node, works:: + + >>> sel.xpath("//a[contains(., 'Next Page')]").getall() + ['Click here to go to the Next Page'] + +.. _`XPath string function`: https://www.w3.org/TR/xpath/#section-String-Functions + .. _topics-selectors-xpath-variables: Variables in XPath expressions @@ -461,6 +632,69 @@ on `XPath variables`_. .. _XPath variables: https://parsel.readthedocs.io/en/latest/usage.html#variables-in-xpath-expressions + +.. _removing-namespaces: + +Removing namespaces +------------------- + +When dealing with scraping projects, it is often quite convenient to get rid of +namespaces altogether and just work with element names, to write more +simple/convenient XPaths. You can use the +:meth:`Selector.remove_namespaces` method for that. + +Let's show an example that illustrates this with GitHub blog atom feed. + +.. highlight:: sh + +First, we open the shell with the url we want to scrape:: + + $ scrapy shell https://github.com/blog.atom + +.. highlight:: xml + +This is how the file starts:: + + + + tag:github.com,2008:/blog + ... + +You can see two namespace declarations: a default "http://www.w3.org/2005/Atom" +and another one using the "media:" prefix for "http://search.yahoo.com/mrss/". + +.. highlight:: python + +Once in the shell we can try selecting all ```` objects and see that it +doesn't work (because the Atom XML namespace is obfuscating those nodes):: + + >>> response.xpath("//link") + [] + +But once we call the :meth:`Selector.remove_namespaces` method, all +nodes can be accessed directly by their names:: + + >>> response.selector.remove_namespaces() + >>> response.xpath("//link") + [, + , + ... + +If you wonder why the namespace removal procedure isn't always called by default +instead of having to call it manually, this is because of two reasons, which, in order +of relevance, are: + +1. Removing namespaces requires to iterate and modify all nodes in the + document, which is a reasonably expensive operation to perform by default + for all documents crawled by Scrapy + +2. There could be some cases where using namespaces is actually required, in + case some element names clash between namespaces. These cases are very rare + though. + + Using EXSLT extensions ---------------------- @@ -606,174 +840,44 @@ inside another ``itemscope``. .. _regular expressions: http://exslt.org/regexp/index.html .. _set manipulation: http://exslt.org/set/index.html +Other XPath extensions +---------------------- -Some XPath tips ---------------- +Scrapy selectors also provide a sorely missed XPath extension function +``has-class`` that returns ``True`` for nodes that have all of the specified +HTML classes. -Here are some tips that you may find useful when using XPath -with Scrapy selectors, based on `this post from ScrapingHub's blog`_. -If you are not much familiar with XPath yet, -you may want to take a look first at this `XPath tutorial`_. +.. highlight:: html +For the following HTML:: -.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html -.. _`this post from ScrapingHub's blog`: https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/ +

        First

        +

        Second

        +

        Third

        +

        Fourth

        +.. highlight:: python -Using text nodes in a condition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +You can use it like this:: -When you need to use the text content as argument to an `XPath string function`_, -avoid using ``.//text()`` and use just ``.`` instead. - -This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*. -And when a node-set is converted to a string, which happens when it is passed as argument to -a string function like ``contains()`` or ``starts-with()``, it results in the text for the first element only. - -Example:: - - >>> from scrapy import Selector - >>> sel = Selector(text='Click here to go to the Next Page') - -Converting a *node-set* to string:: - - >>> sel.xpath('//a//text()').getall() # take a peek at the node-set - ['Click here to go to the ', 'Next Page'] - >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string - ['Click here to go to the '] - -A *node* converted to a string, however, puts together the text of itself plus of all its descendants:: - - >>> sel.xpath("//a[1]").getall() # select the first node - ['Click here to go to the Next Page'] - >>> sel.xpath("string(//a[1])").getall() # convert it to string - ['Click here to go to the Next Page'] - -So, using the ``.//text()`` node-set won't select anything in this case:: - - >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall() + >>> response.xpath('//p[has-class("foo")]') + [, + ] + >>> response.xpath('//p[has-class("foo", "bar-baz")]') + [] + >>> response.xpath('//p[has-class("foo", "bar")]') [] -But using the ``.`` to mean the node, works:: +So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS +``p.foo.bar-baz``. Please note, that it is slower in most of the cases, +because it's a pure-Python function that's invoked for every node in question +whereas the CSS lookup is translated into XPath and thus runs more efficiently, +so performance-wise its uses are limited to situations that are not easily +described with CSS selectors. - >>> sel.xpath("//a[contains(., 'Next Page')]").getall() - ['Click here to go to the Next Page'] +Parsel also simplifies adding your own XPath extensions. -.. _`XPath string function`: https://www.w3.org/TR/xpath/#section-String-Functions - -Beware of the difference between //node[1] and (//node)[1] -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -``//node[1]`` selects all the nodes occurring first under their respective parents. - -``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them. - -Example:: - - >>> from scrapy import Selector - >>> sel = Selector(text=""" - ....:
          - ....:
        • 1
          • - ....:
        • 2
          • - ....:
        • 3
          • - ....:
        - ....:
          - ....:
        • 4
          • - ....:
        • 5
          • - ....:
        • 6
          • - ....:
        """) - >>> xp = lambda x: sel.xpath(x).getall() - -This gets all first ``
      • `` elements under whatever it is its parent:: - - >>> xp("//li[1]") - ['
      • 1
        • ', '
      • 4
        • '] - -And this gets the first ``
      • `` element in the whole document:: - - >>> xp("(//li)[1]") - ['
      • 1
        • '] - -This gets all first ``
      • `` elements under an ``
          `` parent:: - - >>> xp("//ul/li[1]") - ['
        • 1
          • ', '
        • 4
          • '] - -And this gets the first ``
        • `` element under an ``
            `` parent in the whole document:: - - >>> xp("(//ul/li)[1]") - ['
          • 1
            • '] - -When querying by class, consider using CSS -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Because an element can contain multiple CSS classes, the XPath way to select elements -by class is the rather verbose:: - - *[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')] - -If you use ``@class='someclass'`` you may end up missing elements that have -other classes, and if you just use ``contains(@class, 'someclass')`` to make up -for that you may end up with more elements that you want, if they have a different -class name that shares the string ``someclass``. - -As it turns out, Scrapy selectors allow you to chain selectors, so most of the time -you can just select by class using CSS and then switch to XPath when needed:: - - >>> from scrapy import Selector - >>> sel = Selector(text='
            ') - >>> sel.css('.shout').xpath('./time/@datetime').getall() - ['2014-07-23 19:00'] - -This is cleaner than using the verbose XPath trick shown above. Just remember -to use the ``.`` in the XPath expressions that will follow. - -.. _old-extraction-api: - -extract() and extract_first() ------------------------------ - -If you're a long-time Scrapy user, you're probably familiar -with ``.extract()`` and ``.extract_first()`` selector methods. These methods -are still supported by Scrapy, there are no plans to deprecate them. - -However, Scrapy usage docs are now written using ``.get()`` and -``.getall()`` methods. We feel that these new methods result in a more concise -and readable code. - -The following examples show how these methods map to each other. - -1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``:: - - >>> response.css('a::attr(href)').get() - 'image1.html' - >>> response.css('a::attr(href)').extract_first() - 'image1.html' - -2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``:: - - >>> response.css('a::attr(href)').getall() - ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] - >>> response.css('a::attr(href)').extract() - ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html'] - -2. ``Selector.get()`` is the same as ``Selector.extract()``:: - - >>> response.css('a::attr(href)')[0].get() - 'image1.html' - >>> response.css('a::attr(href)')[0].extract() - 'image1.html' - -4. For consistency, there is also ``Selector.getall()``, which returns a list:: - - >>> response.css('a::attr(href)')[0].getall() - ['image1.html'] - -So, the main difference is that output of ``.get()`` and ``.getall()`` methods -is more predictable: ``.get()`` always returns a single result, ``.getall()`` -always returns a list of all extracted results. With ``.extract()`` method -it was not always obvious if a result is a list or not; to get a single -result either ``.extract()`` or ``.extract_first()`` should be called. +.. autofunction:: parsel.xpathfuncs.set_xpathfunc .. _topics-selectors-ref: @@ -909,6 +1013,11 @@ SelectorList objects their results flattened, as a list of unicode strings. +.. _selector-examples: + +Examples +======== + .. _selector-examples-html: Selector examples on HTML response @@ -958,65 +1067,4 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:: sel.register_namespace("g", "http://base.google.com/ns/1.0") sel.xpath("//g:price").getall() -.. _removing-namespaces: - -Removing namespaces -------------------- - -When dealing with scraping projects, it is often quite convenient to get rid of -namespaces altogether and just work with element names, to write more -simple/convenient XPaths. You can use the -:meth:`Selector.remove_namespaces` method for that. - -Let's show an example that illustrates this with GitHub blog atom feed. - -.. highlight:: sh - -First, we open the shell with the url we want to scrape:: - - $ scrapy shell https://github.com/blog.atom - -.. highlight:: xml - -This is how the file starts:: - - - - tag:github.com,2008:/blog - ... - -You can see two namespace declarations: a default "http://www.w3.org/2005/Atom" -and another one using the "media:" prefix for "http://search.yahoo.com/mrss/". - -.. highlight:: python - -Once in the shell we can try selecting all ```` objects and see that it -doesn't work (because the Atom XML namespace is obfuscating those nodes):: - - >>> response.xpath("//link") - [] - -But once we call the :meth:`Selector.remove_namespaces` method, all -nodes can be accessed directly by their names:: - - >>> response.selector.remove_namespaces() - >>> response.xpath("//link") - [, - , - ... - -If you wonder why the namespace removal procedure isn't always called by default -instead of having to call it manually, this is because of two reasons, which, in order -of relevance, are: - -1. Removing namespaces requires to iterate and modify all nodes in the - document, which is a reasonably expensive operation to perform by default - for all documents crawled by Scrapy - -2. There could be some cases where using namespaces is actually required, in - case some element names clash between namespaces. These cases are very rare - though. - .. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799