diff --git a/docs/_static/xpathdemo.css b/docs/_static/xpathdemo.css index f31c4e279..e8d954ce2 100644 --- a/docs/_static/xpathdemo.css +++ b/docs/_static/xpathdemo.css @@ -26,6 +26,7 @@ min-height: 25px; padding: 4px 8px; border-radius: 5px; + white-space: pre; } .result_node + .result_node { diff --git a/docs/topics/xpath-tutorial.rst b/docs/topics/xpath-tutorial.rst index 6b85605df..76c924f7c 100644 --- a/docs/topics/xpath-tutorial.rst +++ b/docs/topics/xpath-tutorial.rst @@ -2,27 +2,2025 @@ XPath Tutorial ============== -Welcome to the xpath tutorial. Have a look at this expression: +Part 1: What is XPath? +====================== -.. xpathdemo:: //h2/a +XPath is a language +------------------- + +.. epigraph:: + + *"XPath is a language for addressing parts of an XML document"* + + -- `XML Path Language 1.0 `__ + +This abstract from the official specifications says it all: + +- *"XPath is a language"*: an XPath expression is a character string... +- *"for addressing parts of an XML document"*: ...a string that that + you pass to an XPath engine acting over an XML (or HTML) document, + outputting parts of it, and following the data model explained below. + +Why learn XPath? +---------------- + +- with XPath, you can navigate **everywhere** inside a DOM tree +- it's a must-have skill for accurate web data extraction +- XPath is more powerful than CSS selectors +- it allows selection and filtering with a fine-grained look at the + text content +- XPath allows complex conditioning with axes +- XPath is extensible with custom functions (we won’t cover that in + this tutorial though) + +XPath data model +---------------- + +XPath's `data model `__ is a +tree of nodes representing a document. Nodes can be either: + +- **element nodes** (``

This is a paragraph

``), +- or **attribute nodes** (``href="page.html"`` inside an ```` tag), +- or **text nodes** (``"I have something to say"``), +- or **comment nodes** (````), +- (or root nodes, or namespace nodes, or processing instructions nodes + but we will not cover them here.) + +In XPath's data model, everything is a node : elements, attributes, +comments... (**but not all nodes are elements.**) + +And nodes have an order, the **document order**: the order in which they +appear in the XML/HTML source. + +In effect, this data model allows you to represent everything inside an +XML or HTML document, in a structured, ordered and hierarchical way. + +Throughout this tutorial, we'll use the following sample HTML page to +illustrate how XPath works: + +:: - - My page - - -

Welcome to my page

-

This is the first paragraph

. - - + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ -And this one :-) : +Here is an ASCII tree representation of our toy HTML document for an +XPath engine, according to the data model: -.. xpathdemo:: //h2 +:: + + # 0--(ROOT) + +-- # 1-- + +-- # 2--(TXT): '\n' + +-- # 3-- + | +-- # 4--(TXT): '\n ' + | +-- # 5-- + | | +-- # 6--(TXT): 'This is a title' + | +-- # 7--(TXT): '\n ' + | +-- # 8--<meta> + | | +-- # 9--(ATTR): content: 'text/html; charset=utf-8' + | | +-- #10--(ATTR): http-equiv: 'content-type' + | +-- #11--(TXT): '\n' + +-- #12--(TXT): '\n' + +-- #13--<body> + | +-- #14--(TXT): '\n ' + | +-- #15--<div> + | | +-- #16--(TXT): '\n ' + | | +-- #17--<div> + | | | +-- #18--(TXT): '\n ' + | | | +-- #19--<p> + | | | | +-- #20--(TXT): 'This is a paragraph.' + | | | +-- #21--(TXT): '\n ' + | | | +-- #22--<p> + | | | | +-- #23--(TXT): 'Is this ' + | | | | +-- #24--<a> + | | | | | +-- #25--(ATTR): href: 'page2.html' + | | | | | +-- #26--(TXT): 'a link' + | | | | +-- #27--(TXT): '?' + | | | +-- #28--(TXT): '\n ' + | | | +-- #29--<br> + | | | +-- #30--(TXT): '\n Apparently.\n ' + | | +-- #31--(TXT): '\n ' + | | +-- #32--<div> + | | | +-- #33--(ATTR): class: 'second' + | | | +-- #34--(TXT): '\n Nothing to add.\n Except maybe this ' + | | | +-- #35--<a> + | | | | +-- #36--(ATTR): href: 'page3.html' + | | | | +-- #37--(TXT): 'other link' + | | | +-- #38--(TXT): '. \n ' + | | | +-- #39--(COMM): ' And this comment ' + | | | +-- #40--(TXT): '\n ' + | | +-- #41--(TXT): '\n ' + | +-- #42--(TXT): '\n' + +-- #43--(TXT): '\n' + +You can see various tree branches and leaves: + +- e.g. ``<div>`` or ``<p>``: these are element nodes +- ``(TXT)`` represent text nodes +- ``(ATTR)`` represent attribute nodes +- ``(COMM)`` represent comment nodes + +The ``#<number>`` are the document orders of each node. + +.. note:: + You can also notice that **text with only whitespace** (space and + newlines in our example) **are proper nodes**, they do have their + document order and can be selected with XPath. + +In-browser widget and using parsel +---------------------------------- + +To illustrate and learn XPath, we will use an in-browser widget +allowing you to play around with XPath expressions and see the output +live. +We will also illustrate some Python pattern for data extraction with +XPath using the `parsel <https://github.com/scrapy/parsel>`__ library +which powers Scrapy selectors under the hood. +It is a Python module written on top of `lxml <http://lxml.de/>`__. + +.. note:: + lxml itself is built using the C library `libxml2 <http://www.xmlsoft.org/>`__, + which has a conformant XPath 1.0 engine. + You should be able to run the same XPath expressions with + any XPath 1.0 engine, and get the same results. + +This tutorial only showcases XPath 1.0. (`XPath has reached version 3 +<https://www.w3.org/TR/xpath-3/>`__, but you can already do a +lot with XPath 1.0 and Python. And there's no XPath>1.0 implementation +in Python today.) + +When showing Python code snippets using Parsel, we assume that we have +a ``Selector`` -- called ``doc`` -- created with the HTML content, similarly +to the following: + +.. code:: python + + import parsel + + + htmlsample = '''<html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + ''' + + doc = parsel.Selector(text=htmlsample) + +XPath return types +------------------ + +When applied over a document, an XPath expression can return either: + +- a node-set -- this is the most common case, and often it's a set of + element nodes +- a string +- a number (floating point) +- a boolean + +.. note:: + **When an XPath expression returns a node-set, you do get a set of + nodes, even if there's only one node in the set.** + With parsel, you get a ``list`` of nodes though, not a Python ``set``. + +XPath expressions +----------------- + +We will now take a look at some example XPath expressions to get a +feeling of how they work. We'll explain the syntax in more details later +on. + +XPath expressions are passed to an XPath engine as strings. + + +Selecting the root node of a document (warning: special case) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The root node is a special node: this is a quote from XPath 1.0 specs: + + *"The root node is the root of the tree. A root node does not occur + except as the root of the tree. The element node for the document + element is a child of the root node."* + +Selecting the root node of a document with XPath is one of the shortest +XPath expressions: ``"/"`` (a string with only a forward slash). + +.. xpathdemo:: / -

Welcome to my page

-

This is the first paragraph

. - + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + +This is very similar to ``cd /`` in a shell within a Unix filesystem +(going to the root directory). + +.. warning:: + Unfortunately, this ``"/"`` expression does not work as expected + with parsel. We get an empty list instead of the root node. + + It is a limitation of lxml apparently, because + it works with libxml2 directly. In practice though, this doesn't matter + much because the root node is virtually never used directly. + + +Selecting elements +~~~~~~~~~~~~~~~~~~ + +Elements build the structure and hierarchy of the document. An element +in HTML (and XML) is what you see in the source code between an opening +and corresponding closing tag, and everything in between. + +- ``This is a title`` is a ``title`` element, +- ``

Is this a link?

`` is a ``p`` + (paragraph) element. + +Selecting elements is probably the most common use-case for XPath on +HTML documents. + +Elements can have children -- the root node being the ancestor of them +all. Their children can also have children and so on. Sometimes, +elements only have one child. This hierarchy forms a family tree of nodes. + +.. note:: + **Text nodes are not elements.** (They are still nodes, obviously.) + They do not have children nodes, but they are always children + of some element. + + Therefore, text nodes are always leaves of the document tree. + +We said earlier that the document element is a child of the root node. +In fact, the document element is the only child of the root node. And +for our sample HTML document, it's the top-level ``...`` element. +Still, selecting it will return a single-node node-set, the XPath expression +being ``/*``: + +.. xpathdemo:: /* + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +The asterisk here, ``*``, means "any element". And ``/*`` means "any +element under the root node". HTML documents have only one element like +this: the ```` element. + +Another example: how to get ```` elements? Use ``/html/head/title``: + +.. xpathdemo:: /html/head/title + + <html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +Again, if you are familiar with the Unix filesystem, you probably +intuitively understand what this does: + +* start from the root (of the document) + + * select the ```` node (with ``/html``) + + * select the ```` node under the ```` node + (appending ``/head``) + + * select the ```` node under the ``<head>`` node + (appending ``/title``) + +In other words, the XPath expression represents the path from the root +node down to the target node(s). Parts of this path are read **from left to right**, +and represent a top-to-bottom direction in the document tree. + +Much like a Unix filepath represents the path from the filesystem's root +to the target file(s) or directory(ies). +There's one major difference with a Unix filesystem though: in an HTML +or XML document, an element can have multiple children with the same name. +For example, the ``<div>`` just under the ``<body>`` has 2 ``<div>`` children: + +.. xpathdemo:: /html/body/div/div + + <html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +Another example is getting the paragraphs inside the first child of that +``
`` under ````, there are two of them: + +.. xpathdemo:: /html/body/div/div[1]/p + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +Here we're introducing a **positional predicate**, ``[1]``. The ``div[1]`` +part means *"the first
child under its parent"*. + +If you recall, earlier we used a ``*`` asterisk to mean *any element*. +There are other elements with those two paragraphs under that very +``
``. Let's try and select all of them, regardless of their name: + +.. xpathdemo:: /html/body/div/div[1]/* + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +.. note:: + Continuing the filesystem anamogy, ``*`` is similar in effect to what + you can do in a Unix shell to find files or directories without explicit + full names. + +See the ``
`` being selected? It's an empty element (i.e. with node children) +but it is there nonetheless. + +Selecting text nodes +~~~~~~~~~~~~~~~~~~~~ + +If we stay around these ``

`` and ``
`` elements, you may have noticed +that the ASCII tree representation from the beginning also shows some text after the +``
`` break: the string ``"Apparently."``. It is a text node. + +Selecting text nodes is a bit different than selecting elements: +you use the special ``text()`` syntax. Let's try it by replacing the last +part of our last XPath expression, forming ``/html/body/div/div[1]/text()``: + +.. xpathdemo:: /html/body/div/div[1]/text() + + + + This is a title + + + +

+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +You may have expected only one text result, the last one, ``"Apparently."``. +But we got four! And three of them are blank even. Why is that? + +In fact, HTML authors usually indent their tags with whitespace for +readability. This does not usually change the layout in your browser. +But this **whitespace counts as text nodes** for XPath's data model, +it is not stripped nor filtered. + +Let's represent that ``
`` as a Python string as it appears in the +HTML source:: + + # + # text node #1 text node #2 text node #3 + # <------> <------> <------> + '
\n

This is a paragraph.

\n

Is this a link?

\n
\n Apparently.\n
' + + +We've marked the first three text nodes before the non-whitespace only +text node. + +Another example is to get the text nodes of ```` elements +(remember that ``<title>`` is an element, and that it happens it +contains a text node, with the string content "This is a title"): + +.. xpathdemo:: /html/head/title/text() + + <html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +.. note:: + Again, there's only one ````, and it contains only one text node, + but selecting text nodes in ``<title>`` returns a single string-value + in a list, not one string. + + +Selecting nodes without a full, explicit path +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +What we did until now is tell the XPath engine how to get to nodes, +node by node, from parent to child, from root node down to target nodes. +This assumes that you know the hierarchy of nodes beforehand. +This *can* be the case, but most often than not, +either you do not know or you do not want to indicate all the steps from +the root node down to the node(s) you are interested in (this can be +very error prone -- have you put enough ``div/div/div...``?). + +XPath provides a handy shortcut when you do not know at what level you +expect your target node to be. +Say for example that we want to select all ``<p>`` paragraph elements +inside the ``<body>``. We don't *a-priori* know what their parent node is. +(For all we know, they can be anywhere under the ``<body>`` element.) +The shortcut to use is ``//`` (two forward slashes). +Let's try this: ``//body//p`` + +.. xpathdemo:: //body//p + + <html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +So we got 2 paragraphs, what we expected. + +This also works for text nodes (there are a lot of them in our sample +document!). Try ``//body//text()``: + +.. xpathdemo:: //body//text() + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +Selecting attributes +~~~~~~~~~~~~~~~~~~~~ + +Elements can also have attributes. +In our sample document, we have two ```` elements, each with a +``href`` attribute. There's also a ```` element with two +attributes: ``content`` and ``http-equiv``. + +This is how you can select these attributes, with an ``@`` prefix before +the attribute name: + +.. xpathdemo:: //a/@href + + + + This is a title + + + +
+ +
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +.. xpathdemo:: //meta/@* + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +The ``*`` (asterisk) here after ``@`` means the same thing as in ``/*`` +exept that this is for attributes, and not elements: meaning that you +want any attributes, whatever their name. + + +Get a string representation of an element +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The XPath language also comes with a few string functions, that you can +wrap around an XPath expression selecting elements: + +.. xpathdemo:: string(/html/head/title) + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +This example uses ``string()``, one of several handy +`functions `__ in XPath. +``string()`` will concatenate all text content from the selected node +and all of its children, recursively, effectively stripping HTML tags. + +You may wonder what's the difference between ``string(/html/head/title)`` +and ``/html/head/title/text()`` from earlier? Here, in fact, you get the same +result because ```` only has one child text node. +(Concatenating this list of one text node is the same as getting it +directly with ``text()`` at the end.) + +But string functions can be very handy when you apply them on nodes that +have multiple children and multiple text node children or descendant. +What happens when you apply ``string()`` on the document ``<body>`` for example? +You get a text representation of the document, without the tags: + + +.. xpathdemo:: string(string(//body)) + + <html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +Counting elements +~~~~~~~~~~~~~~~~~ + +We said earlier that XPath expressions could also return numbers. +One example of this is counting the number of paragraphs in the +document: + +.. xpathdemo:: count(//p) + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +.. note:: + With parsel, you get a floating point number back, and in the form of a + string. This is specific to parsel. Another XPath engine might return a + native floating point number. + +Another example: get the number of attributes in the document (whatever +their parent element): + + +.. xpathdemo:: count(//@*) + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + + +Boolean operations +~~~~~~~~~~~~~~~~~~ + +XPath expressions can also return booleans. This is not that usueful +by itself, but it becomes handy when used in predicates (that we will +cover a bit later). + +For example, testing the number of paragraphs: + + +.. xpathdemo:: count(//p) = 2 + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +.. xpathdemo:: count(//p) = 42 + + + + This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + + +Part 2: Location Paths: how to move inside the document tree +============================================================ + +A **Location path** is the most common XPath expression. + +It is used to move in any direction from a starting point (*the context +node*) to any node(s) in the tree: + +- It is a string, with a series of “location steps”: + ``"step1 / step2 / step3 ..."`` +- It represents the selection and filtering of nodes, processed step by + step, from left to right. +- Each step is of the form: ``AXIS :: NODETEST [PREDICATE]*`` + +So the examples we saw earlier are or contain an XPath expression: +``/html/head/title``, ``//body//p`` etc. + +**Note:** whitespace does NOT matter, except for ``“//”`` and ``“..”`` +(``“/ /”`` and ``“. .”`` are syntax errors.). The following 3 +expressions produce the same result + +.. code:: python + + doc.xpath('/html/head/title') + + + + +.. parsed-literal:: + + [(1) 'This is a title'] + + + +.. code:: python + + doc.xpath('/ html / head /title') + + + + +.. parsed-literal:: + + [(1) 'This is a title'] + + + +.. code:: python + + doc.xpath(''' + /html + /head + /title''') + + + + +.. parsed-literal:: + + [(1) 'This is a title'] + + + +So **don’t be afraid of indenting your XPath expressions to improve +readability.** + +Relative vs. absolute paths +--------------------------- + +Location paths can be relative or absolute: + +- ``"step1/step2/step3"`` is relative +- ``"/step1/step2/step3"`` is absolute + +i.e. an absolute path is a relative path starting with "/" (slash) + +In other terms, absolute paths are relative to the root node. + +**Tip**: use relative paths whenever possible. This prevents unexpected +selection of same nodes in loop iterations. + +For example, in our sample document, only one ``
`` contains +paragraphs. Looping on each ``
`` and using the absolute location +path ``//p`` will produce the same result for each iteration: returning +ALL paragraphs in the document everytime. + +.. code:: python + + for div in doc.xpath('//body//div'): + print(div.xpath('//p')) + + +.. parsed-literal:: + + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + + +Compare this with using the relative ``'p'`` or ``'./p'`` expression +that will only look at children ``

`` under each ``

``, and only +one of those ``
`` will show having paragraphs as shown below: + +.. code:: python + + for div in doc.xpath('//body//div'): + print(div.xpath('p')) + + +.. parsed-literal:: + + [] + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + [] + + +.. code:: python + + for div in doc.xpath('//body//div'): + print(div.xpath('./p')) + + +.. parsed-literal:: + + [] + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + [] + + +Abbreviated syntax +------------------ + +What we’ve seen earlier is in fact the “abbreviated syntax” for XPath +expressions. + +The full syntax is quite verbose (but you sometimes need it): + ++-----------------------------+----------------------------------------------+ +| Abbreviated syntax | Full syntax | ++=============================+==============================================+ +| ``/html/head/title`` | ``/child::html /child:: head /child:: title` | +| | ` | ++-----------------------------+----------------------------------------------+ +| ``//meta/@content`` | ``/descendant-or-self::node() /child::meta / | +| | attribute::content`` | ++-----------------------------+----------------------------------------------+ +| ``//div/div[@class="second" | ``/descendant-or-self::node() /child::div /c | +| ]`` | hild::div [attribute::class = "second"]`` | ++-----------------------------+----------------------------------------------+ +| ``//div/a/text()`` | ``/descendant-or-self::node() /child::div /c | +| | hild::a /child::text()`` | ++-----------------------------+----------------------------------------------+ + +What are these ``child::``, ``descendant-or-self::`` and +``attribute::``, you may ask? They're axes. + +Axes: moving around +------------------- + +Remember: each step of an XPath location path is of the form +``AXIS :: NODETEST [PREDICATE]*``. + +The "axis" is the first part of each location path step. It can be +explicit, or implicit in abbreviated syntax. For example, in +``/html/head/title``, the ``child::`` axis is omitted in each step. + +In this section, we'll use explicit axes as much as we can. + +**AXIS** :: \_nodetest [predicate]\*\_ + +**Axes give the direction to go next, one location step at a time** + +- ``self`` (where you are) +- ``parent``, ``child`` (direct hop) +- ``ancestor``, ``ancestor-or-self``, ``descendant``, + ``descendant-or-self`` (multi-hop) +- ``following``, ``following-sibling``, ``preceding``, + ``preceding-sibling`` (document order) +- ``attribute``, ``namespace`` (non-element) + +Move up or down the tree: self, child, descendant, parent, ancestor +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Let's assume that we have selected the first ``
`` element in our +sample document, the one just under the ```` element: + +.. code:: python + + first_div = doc.xpath('//body/div')[0] + first_div + + + + +.. parsed-literal:: + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + + +The ``self`` axis represents *the context node*, i.e. where you are +currently in the Location Path step. (This may not sounds very useful, +but we will see later when this can be handy.) + +.. code:: python + + first_div.xpath('self::*') + + + + +.. parsed-literal:: + + [(1) '
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
'] + + + +If you chain ``self::`` steps, you'll stay on the same context node: + +.. code:: python + + first_div.xpath('self::*/self::*/self::*') + + + + +.. parsed-literal:: + + [(1) '
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
'] + + + +``self::`` is usually seen in abbreviated form: i.e. a '.' (dot). So you +could aslo use: + +.. code:: python + + first_div.xpath('.') + + + + +.. parsed-literal:: + + [(1) '
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
'] + + + +.. code:: python + + first_div.xpath('././.') + + + + +.. parsed-literal:: + + [(1) '
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
'] + + + +The ``child`` axis is for immediate children nodes of the context node. +Here, our context ``
`` node has 2 ``
`` children: + +.. code:: python + + first_div.xpath('child::*') + + + + +.. parsed-literal:: + + [(1) '
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
' + (2) '
+ Nothing to add. + Except maybe this other link. + +
'] + + + +``child`` is in fact the default axis, hence it can be omitted (e.g. we +saw that ``/html/head/title`` is equivalent of +``/child::html/child::head/child::title``.) + +The ``parent`` axis is the dual of ``child``: you go up one level in the +DOM: + +.. code:: python + + first_div.xpath('parent::*') + + + + +.. parsed-literal:: + + [(1) ' +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ '] + + + +There's an alias for ``parent::``: it's ``..`` (2 dots, much like in a +Unix filesystem): + +.. code:: python + + first_div.xpath('..') + + + + +.. parsed-literal:: + + [(1) ' +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ '] + + + +Let's simplify our ASCII tree representation from earlier to only +consider element nodes: + +:: + + # 0--(ROOT) + +-- # 1-- + +-- # 3-- + | +-- # 5-- + | +-- # 8--<meta> + +-- #13--<body> + +-- #15--<div> + +-- #17--<div> + | +-- #19--<p> + | +-- #22--<p> + | | +-- #24--<a> + | +-- #29--<br> + +-- #32--<div> + +-- #35--<a> + +With this simplified tree representation, this is what ``self``, +``child`` and ``parent`` select: + +:: + + # 0--(ROOT) + +-- # 1--<html> + +-- # 3--<head> + | +-- # 5--<title> + | +-- # 8--<meta> + parent::* ---------> +-- #13--<body> + | + self::* -------------> +-- #15--<div> + | + child::*----+-----------> +-- #17--<div> + | | +-- #19--<p> + | | +-- #22--<p> + | | | +-- #24--<a> + | | +-- #29--<br> + +-----------> +-- #32--<div> + +-- #35--<a> + +Recursively go up or down +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``descendant`` axis is similar to ``child`` but also goes deeper in +the tree, looking at children of each child, recursively: + +.. code:: python + + first_div.xpath('descendant::*') + + + + +.. parsed-literal:: + + [(1) '<div> + <p>This is a paragraph.</p> + <p>Is this <a href="page2.html">a link</a>?</p> + <br> + Apparently. + </div>' + (2) '<p>This is a paragraph.</p>' + (3) '<p>Is this <a href="page2.html">a link</a>?</p>' + (4) '<a href="page2.html">a link</a>' + (5) '<br>' + (6) '<div class="second"> + Nothing to add. + Except maybe this <a href="page3.html">other link</a>. + <!-- And this comment --> + </div>' + (7) '<a href="page3.html">other link</a>'] + + + +You might guess already what ``ancestor`` is for: it is the dual axis of +``descendant``: + +.. code:: python + + first_div.xpath('ancestor::*') + + + + +.. parsed-literal:: + + [(1) '<html> + <head> + <title>This is a title + + + +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ + ' + (2) ' +
+
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
+
+ Nothing to add. + Except maybe this other link. + +
+
+ '] + + + +Special case of ``descendant-or-self`` axis +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TODO: explain + +.. code:: python + + first_div.xpath('./descendant-or-self::node()/text()') + + + + +.. parsed-literal:: + + [(1) ' + ' + (2) ' + ' + (3) 'This is a paragraph.' + (4) ' + ' + (5) 'Is this ' + (6) 'a link' + (7) '?' + (8) ' + ' + (9) ' + Apparently. + ' + (10) ' + ' + (11) ' + Nothing to add. + Except maybe this ' + (12) 'other link' + (13) '. + ' + (14) ' + ' + (15) ' + '] + + + +Move "sideways": children nodes of the same parent +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If nodes can have parents, children, ancestors and descendants, they can +also have siblings (to continue the family metaphor). **Siblings are +nodes that have the same parent node.** + +Some siblings may come before the context node (they appear before in +the document, their order is lower), or they can come after the context +node. There are 2 axis for these 2 directions: ``preceding-sibling`` and +``following-sibling``. + +Let's first select this paragraph from our sample document: +``

Is this a link?

``. It's the 2nd child +of the 1st ``
`` of the ``
`` we used above: + +.. code:: python + + paragraph = first_div.xpath('child::div[1]/child::p[2]')[0] + +You can notice above that we started using 2 new patterns along with the +axes: + +- ``child::div`` vs. ``child::*``: ``*`` means "any element node" (this + is a *NODETEST* that we'll cover afterwards) +- ``[1]`` and ``[2]``: which mean *first* and *second* in the current + step's node-set (this is a kind of *PREDICATE* that we'll cover + afterwards also) + +.. code:: python + + paragraph.xpath('preceding-sibling::*') + + + + +.. parsed-literal:: + + [(1) '

This is a paragraph.

'] + + + +.. code:: python + + paragraph.xpath('following-sibling::*') + + + + +.. parsed-literal:: + + [(1) '
'] + + + +Again, let's see which elements were selected in our ASCII tree +representation: + +:: + + # 0--(ROOT) + +-- # 1-- + +-- # 3-- + | +-- # 5-- + | +-- # 8--<meta> + +-- #13--<body> + | + +-- #15--<div> + | + +-- #17--<div> + | | + | | + preceding-sibling::* ----------> +-- #19--<p> + | | + | | + self::* -----------------------> +-- #22--<p> + | | | + | | +-- #24--<a> + | | + | | + following-sibling::* ----------> +-- #29--<br> + | + | + +-- #32--<div> + +-- #35--<a> + +.. code:: python + + paragraph.xpath('following-sibling::node()') + + + + +.. parsed-literal:: + + [(1) ' + ' + (2) '<br>' + (3) ' + Apparently. + '] + + + +Nodes before and after, in document order +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``preceding`` and ``following`` are 2 special axes that do not look at +the tree hierarchy, but work on the document order of nodes. + +Remember, all nodes in XPath data model have an order, called the +*document order*. Node 1 is the first node in the HTML source, node 2 is +the node appearing next etc. + +:: + + #1 #2 #3 ... + <html><head><title>... + +.. code:: python + + paragraph.xpath('preceding::*') + + + + +.. parsed-literal:: + + [(1) '<head> + <title>This is a title + + ' + (2) 'This is a title' + (3) '' + (4) '

This is a paragraph.

'] + + + +.. code:: python + + paragraph.xpath('following::*') + + + + +.. parsed-literal:: + + [(1) '
' + (2) '
+ Nothing to add. + Except maybe this other link. + +
' + (3) 'other link'] + + + +Note that ``preceding`` does not include ancestors and ``following`` +does not include descendants. + +This property is mentioned in XPath specs like this: + + The ancestor, descendant, following, preceding and self axes + partition a document (ignoring attribute and namespace nodes): they + do not overlap and together they contain all the nodes in the + document. + +i.e. +``document == self U (ancestor U preceding) U (descendant U following)`` + +Node tests +---------- + +A "node test" is the second part of each step in a location path. + +*axis* :: **NODETEST** \_[predicate]\*\_ + +Node tests select node types along the step's axis. + +They can be: + +- a *name test*: + +- such as "p", "title" or "a" for elements: ``/html/head/title`` + contains 3 steps, each with a name test node test +- or "href" or "src" for attributes: ``/a/@href`` selects "href" + attributes of + +- a \*node type test": + +- "node()": any node type +- "text()": text nodes +- "comment()": comment nodes +- "\*" (an asterisk): the meaning depends on the axis: + + - an "\*" step alone selects any element nodes (a.k.a tags) + - an "@\*" selects any attribute node + +**Note:** ``text()`` is not a function call that converts a node to it's +text representation, it's just a test on the node type. + +Compare these 2 expressions: + +.. code:: python + + paragraph.xpath('child::text()') + + + + +.. parsed-literal:: + + [(1) 'Is this ' + (2) '?'] + + + +.. code:: python + + paragraph.xpath('string(self::*)') + + + + +.. parsed-literal:: + + [(1) 'Is this a link?'] + + + +``child::text()`` selector all children nodes that are also text nodes. +"a" is part of the ```` inside the paragraph, sot it's not selected. + +Whereas ``string(self::*)`` applies to the paragraph (the context node, +selected with ``self::*``) and recursively gets text content of +children, children of children and so on. + +Predicates +---------- + +*axis* :: *nodetest* \*\* [PREDICATE]\* \*\* + +Predicates are the last part of each step in a location path. Predicates +are optional. + +They are used to further filter nodes on properties that cannot be +expressed with the step's axis and node test. + +Remember that XPath location paths work step by step. Each step produces +a node-set for each node from the previous step's node-set, with +possibly more than 1 node in each node set. + +You may not be interested in all nodes from a node test. + +The syntax for predicates is simple: just surround conditions withing +square brackets. What's inside the square brackets can be: + +- a number (see positional predicates below) +- a location path: the predicate will select nodes for which the + location path matches at least a node +- a boolean operation: for example to test a condition on text content + or count of children + +Positional predicates +~~~~~~~~~~~~~~~~~~~~~ + +The first use-case is selecting nodes based on their position in a +node-set. Node-sets order depends on the axis, but let's consider that +the order of a node in a node-set is the document order. + +Let's say we don't want the 2 paragraphs in the ``
`` we looked at +earlier, only the first one: + +.. code:: python + + doc.xpath('//body/div/div/p') + + + + +.. parsed-literal:: + + [(1) '

This is a paragraph.

' + (2) '

Is this a link?

'] + + + +.. code:: python + + doc.xpath('//body/div/div/p[1]') + + + + +.. parsed-literal:: + + [(1) '

This is a paragraph.

'] + + + +If you want the last node in a node-set, you can use ``last()``: + +.. code:: python + + doc.xpath('//body/div/div[last()]') + + + + +.. parsed-literal:: + + [(1) '
+ Nothing to add. + Except maybe this other link. + +
'] + + + +Position ranges +^^^^^^^^^^^^^^^ + +TODO: things like ``//table/tbody/tr[position() > 2]`` + +Location paths as predicates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO: things like ``//table[tr/div/a]`` + +Boolean predicates +~~~~~~~~~~~~~~~~~~ + +TODO: things like ``//table[count(tr)=10]`` + +Special case of string value tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TODO: things like ``//table[.//img/@src="pic.png"]`` or +``//table[th="Some headers"]`` + +Special trick for testing multiple node names +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TODO: things like ``./descendant-or-self::*[self::ul or self::ol]`` + +Nested predicates +~~~~~~~~~~~~~~~~~ + +We said that location paths can be used as predicate. And location paths +can have predicates. So it's possible end up with nested predicates. + +.. code:: python + + # <------predicate ---------> + # <-nested predicate-> + doc.xpath('//div[p [a/@href="page2.html"] ]') + + + + +.. parsed-literal:: + + [(1) '
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
'] + + + +In fact, the above is equivalent to ``//div[p/a/@href="page2.html"]`` +with no nesting: + +.. code:: python + + doc.xpath('//div[p/a/@href="page2.html"]') + + + + +.. parsed-literal:: + + [(1) '
+

This is a paragraph.

+

Is this a link?

+
+ Apparently. +
'] + + + +Order of predicates is important +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can have multiple predicates in sequence per step, each within its +``[]`` brackets, i.e. steps in the form of +``axis::nodetest[predicate#1][predicate#2][predicate#3]...`` + +Predicates are processed in order, from left to right. And the output of +one predicate is fed into the next predicate filter, much like steps +produce node-sets for the next step to process. + +So the order of predicates is important. + +The following 2 location paths produce different results: + +.. code:: python + + doc.xpath('//div[2][@class="second"]') + + + + +.. parsed-literal:: + + [(1) '
+ Nothing to add. + Except maybe this other link. + +
'] + + + +.. code:: python + + doc.xpath('//div[@class="second"][2]') + + + + +.. parsed-literal:: + + [] + + + +The 2nd produces nothing. Why is that? + +``//div[2][@class="second"]`` looks at ``div`` elements that are the 2nd +child of their parent (because ``div`` means ``child::div``, and ``[2]`` +will select the 2nd node in the current node-set. In our document this +happens only once. The final predicate, ``[@class="second"]``, filter +nodes that have a "class" attribute with value "second". This happens to +be valid for that 2nd child ``div``. + +On the contrary, ``//div[@class="second"][2]`` will first produce +``//div[@class="second"]``, which only produces single-node node-sets +(again, there's only 1 ``div`` with "class" attribute with value +"second"). So the subsequent ``[2]`` predicate will never match with +single-node node-sets. + +Abbreviation cheatsheet +~~~~~~~~~~~~~~~~~~~~~~~ + ++-----------------------------+----------------------------------------------+ +| Abbreviated step | Meaning | ++=============================+==============================================+ +| ``*`` (asterisk) | all **element** nodes (i.e. not text nodes, | +| | not attribute nodes; | ++-----------------------------+----------------------------------------------+ +| | remember that ``.//*`` is not the same as | +| | ``.//node()``; | ++-----------------------------+----------------------------------------------+ +| | also, there's not ``element()`` node test | ++-----------------------------+----------------------------------------------+ +| ``@*`` | ``attribute::*`` (all attribute nodes) | ++-----------------------------+----------------------------------------------+ +| ``//`` | ``/descendant-or-self::node()/ | | +| | | exactly this, nothing mo | +| | re, nothing less, | | +| | | so``//*``is not the same as``/descen | +| | dant-or-self::*\ ``|``.\ ``(a single dot) | +| | |``\ self::node()\ ``, the context n | +| | ode; useful for formation a relative XPath | | +| | | e.g.``.//tr\ | +| | ``|``..\ ``(2 dots) |``\ par | +| | ent::node()\` | ++-----------------------------+----------------------------------------------+ + +TODO: explain why ``//*`` is not the same as ``/descendant-or-self::*`` + +String functions +---------------- + +TODO + +Part 3: Use-cases for web scraping +================================== + +TODO + +Text extraction +--------------- + +TODO + +Attributes extraction +--------------------- + +TODO + +Attribute names extractions +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO + +CSS Selectors +------------- + +TODO + +Loop on elements (table rows, lists) +------------------------------------ + +TODO + +Element boundaries & XPath buckets (advanced) +--------------------------------------------- + +TODO + +EXSLT extensions +---------------- + +TODO + +Summary of tips +=============== + +- Use relative XPath expressions whenever possible +- Know your axes! +- Don't forget that XPath has ``string()`` and ``normalize-space()`` + functions +- **``text()`` is a node test**, not a function call +- CSS selectors are very handy, easier to maintain, but also less + powerful than XPath +