diff --git a/docs/faq.rst b/docs/faq.rst index 490e641cd..5dcb462a2 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -268,6 +268,11 @@ section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument. +I'm scraping a XML document and my XPath selector doesn't return any items +-------------------------------------------------------------------------- + +You may need to remove namespaces. See :ref:`removing-namespaces`. + .. _user agents: http://en.wikipedia.org/wiki/User_agent .. _LIFO: http://en.wikipedia.org/wiki/LIFO .. _DFO order: http://en.wikipedia.org/wiki/Depth-first_search diff --git a/docs/news.rst b/docs/news.rst index 59c7c849c..a0334157a 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -6,7 +6,7 @@ Release notes 0.18 (unreleased) ----------------- -- added :meth:`XmlXPathSelector.remove_namespaces` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`. +- added :meth:`XPathSelector.remove_namespaces` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`. - several improvements to spider contracts - New default middleware named MetaRefreshMiddldeware that handles meta-refresh html tag redirections, MetaRefreshMiddldeware and RedirectMiddleware have different priorities to address #62 diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 54e9a81fc..062308149 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -382,24 +382,47 @@ instantiated with a :class:`~scrapy.http.Response` object like this:: x.register_namespace("g", "http://base.google.com/ns/1.0") x.select("//g:price").extract() +.. _removing-namespaces: + Removing namespaces ~~~~~~~~~~~~~~~~~~~ When dealing with scraping projects, it is often quite convenient to get rid of -namespaces altogether and just work with element names. You can use the -:meth:`XmlXPathSelector.remove_namespaces` method for that. +namespaces altogether and just work with element names, to write more +simple/convenient XPaths. You can use the +:meth:`XPathSelector.remove_namespaces` method for that. -Here's an example that illustrates this to parse the Github blog atom feed:: +Let's show an example that illustrates this with Github blog atom feed. + +First, we open the shell with the url we want to scrape:: $ scrapy shell https://github.com/blog.atom - # ... + +Once in the shell we can try selecting all ```` objects and see that it +doesn't work (because the Atom XML namespace is obfuscating those nodes):: + >>> xxs.select("//link") [] + +But once we call the :meth:`XPathSelector.remove_namespaces` method, all +nodes can be accessed directly by their names:: + >>> xxs.remove_namespaces() >>> xxs.select("//link") [, ... - ] + +If you wonder why the namespace removal procedure is not always called, instead +of having to call it manually. This is because of two reasons which, in order +of relevance, are: + +1. removing namespaces requires to iterate and modify all nodes in the + document, which is a reasonably expensive operation to performs for all + documents crawled by Scrapy + +2. there could be some cases where using namespaces is actually required, in + case some element names clash between namespaces. These cases are very rare + though. .. _Google Base XML feed: http://base.google.com/support/bin/answer.py?hl=en&answer=59461