diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 6fc0eee51..d966a67d2 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -407,6 +407,129 @@ inside another ``itemscope``. .. _regular expressions: http://www.exslt.org/regexp/index.html .. _set manipulation: http://www.exslt.org/set/index.html + +Some XPath tips +--------------- + +Here are some tips that you may find useful when using XPath +with Scrapy selectors, based on `this post from ScrapingHub's blog`_. +If you are not much familiar with XPath yet, +you may want to take a look first at this `XPath tutorial`_. + + +.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html +.. _`this post from ScrapingHub's blog`: http://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/ + + +Using text nodes in a condition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you need to use the text content as argument to a `XPath string function`_, +avoid using ``.//text()`` and use just ``.`` instead. + +This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*. +And when a node-set is converted to a string, which happens when it is passed as argument to +a string function like ``contains()`` or ``starts-with()``, it results in the text for the first element only. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text='Click here to go to the Next Page') + +Converting a *node-set* to string:: + + >>> sel.xpath('//a//text()').extract() # take a peek at the node-set + [u'Click here to go to the ', u'Next Page'] + >>> sel.xpath("string(//a[1]//text())").extract() # convert it to string + [u'Click here to go to the '] + +A *node* converted to a string, however, puts together the text of itself plus of all its descendants:: + + >>> sel.xpath("//a[1]").extract() # select the first node + [u'Click here to go to the Next Page'] + >>> sel.xpath("string(//a[1])").extract() # convert it to string + [u'Click here to go to the Next Page'] + +So, using the ``.//text()`` node-set won't select anything in this case:: + + >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").extract() + [] + +But using the ``.`` to mean the node, works:: + + >>> sel.xpath("//a[contains(., 'Next Page')]").extract() + [u'Click here to go to the Next Page'] + +.. _`XPath string function`: http://www.w3.org/TR/xpath/#section-String-Functions + +Beware the difference between //node[1] and (//node)[1] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``//node[1]`` selects all the nodes occurring first under their respective parents. + +``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text=""" + ....: + ....: """) + >>> xp = lambda x: sel.xpath(x).extract() + +This gets all first ``
  • `` elements under whatever it is its parent:: + + >>> xp("//li[1]") + [u'
  • 1
  • ', u'
  • 4
  • '] + +And this gets the first ``
  • `` element in the whole document:: + + >>> xp("(//li)[1]") + [u'
  • 1
  • '] + +This gets all first ``
  • `` elements under an ``