diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 6fc0eee51..171935c9a 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -407,6 +407,129 @@ inside another ``itemscope``. .. _regular expressions: http://www.exslt.org/regexp/index.html .. _set manipulation: http://www.exslt.org/set/index.html + +Some XPath tips +--------------- + +Here are some tips that you may find useful when using XPath +with Scrapy selectors. If you are not much familiar with XPath yet, +you may want to take a look first at this `XPath tutorial`_. + +You can also find `more XPath tips like these here.`_ + +.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html +.. _`more XPath tips like these here.`: http://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/ + + +Using text nodes in a condition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you need to use the text content as argument to a XPath string function +like ``contains`` or ``substring-after``, avoid using ``.//text()`` and use +just ``.`` instead. + +This is because the ``.//text()`` selects several text nodes (a node-set), +that when converted to string gets only the first element. The ``.`` notation +selects the whole element, which gets the whole element text, +including its descendants. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text='Click here to go to the Next Page') + +Using the ``.//text()`` node-set:: + + >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").extract() + [] + +Using the node:: + + >>> sel.xpath("//a[contains(., 'Next Page')]").extract() + [u'Click here to go to the Next Page'] + +You can verify this difference between the conversion to string from a node or +a *node-set* using the ``string()`` XPath function. + +Converting a *node-set* to string:: + + >>> sel.xpath("string(//a[1]//text())").extract() + [u'Click here to go to the '] + +Converting a *node* to string:: + + >>> sel.xpath("string(//a[1])").extract() + [u'Click here to go to the Next Page'] + + +Beware the difference between //node[1] and (//node)[1] +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``//node[1]`` selects all the nodes occurring first under their respective parents. + +``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them. + +Example:: + + >>> from scrapy import Selector + >>> sel = Selector(text=""" + ....: