RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated Selectors documentation 

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40637
This commit is contained in:
elpolilla 2009-01-04 01:15:08 +00:00
parent a5025899ab
commit ea439e4b03
2 changed files with 35 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
scrapy/trunk/docs/topics/sample1.htm Normal file
View File

@ -0,0 +1,16 @@
<html>
<head>
<base href='http://example.com/' />
<title>Example website</title>
</head>
<body>
<div id='images'>
<a href='image1.html'>Name: My image 1 <br /><img src='image1_thumb.jpg' /></a>
<a href='image2.html'>Name: My image 2 <br /><img src='image2_thumb.jpg' /></a>
<a href='image3.html'>Name: My image 3 <br /><img src='image3_thumb.jpg' /></a>
<a href='image4.html'>Name: My image 4 <br /><img src='image4_thumb.jpg' /></a>
<a href='image5.html'>Name: My image 5 <br /><img src='image5_thumb.jpg' /></a>
</div>
</body>
</html>

37
scrapy/trunk/docs/topics/selectors.rst
View File

@ -1,7 +1,9 @@
.. _topics-selectors:
=========
Selectors
---------
=========
Selectors are *the* way you have to extract information from documents. They retrieve information from the response's body, given an XPath, or a Regular Expression that you provide.
Currently there are two kinds of selectors, HtmlXPathSelectors, and XmlXPathSelectors. Both work in the same way; they are first instanciated with a response, for example::
@ -9,24 +11,22 @@ Currently there are two kinds of selectors, HtmlXPathSelectors, and XmlXPathSele
hxs = HtmlXPathSelector(response) # an HTML selector
xxs = XmlXPathSelector(response) # an XML selector
Now, before going on with selectors, I must tell you about a pretty cool feature that Scrapy has, and which you'll surely find very useful whenever you're writing spiders.
Now, before going on with selectors, I'd suggest you to open a Scrapy shell, which you can use by calling your project manager with the 'shell' argument; something like::
This feature is the Scrapy shell, and you can use it by calling your project manager with the 'shell' argument; something like::
[user@host ~/myproject]$ ./scrapy-manager.py shell <url>
[user@host ~/myproject]$ ./scrapy-ctl.py shell <url>
Notice that you'll have to install IPython in order to use this feature, but believe me that it worths it; the shell is **very** useful.
With the shell you can simulate parsing a webpage, either by calling "scrapy-manager shell" with an url as an additional parameter, or by using the shell's 'get' command, which tries
With the shell you can simulate parsing a webpage, either by calling "scrapy-ctl.py shell" with an url as an additional parameter, or by using the shell's 'get' command, which tries
to retreive the given url, and fills in the 'response' variable with the result.
Ok, so now let's use the shell to show you a bit how do selectors work.
We'll use an example page located in Scrapy's site (http://docs.scrapy.org/examples/sample1.htm), whose markup is::
We'll use an example page located in Scrapy's site (http://www.scrapy.org/docs/topics/sample1.htm), whose markup is::
<html>
<head>
<base href='http://mywebsite.com/' />
<title>My website</title>
<base href='http://example.com/' />
<title>Example website</title>
</head>
<body>
<div id='images'>
@ -41,19 +41,19 @@ We'll use an example page located in Scrapy's site (http://docs.scrapy.org/examp
First, we open the shell::
[user@host ~/myproject]$ ./scrapy-manager.py shell 'http://docs.scrapy.org/examples/sample1.html'
[user@host ~/myproject]$ ./scrapy-ctl.py shell 'http://www.scrapy.org/docs/topics/sample1.htm'
Then, after the shell loads, you'll have some already-made objects for you to play with. Two of them, hxs and xxs, are selectors.
You could instanciate your own by doing::
from scrapy.xpath.selector import HtmlXPathSelector, XmlXPathSelector
my_html_selector = HtmlXPathSelector(r)
my_xml_selector = XmlXPathSelector(r)
my_html_selector = HtmlXPathSelector(response)
my_xml_selector = XmlXPathSelector(response)
Where 'r' is the object that scrapy already created for you containing the given url's response.
Where 'response' is the object that Scrapy already created for you containing the given url's response.
But anyway, we'll stick to the selectors scrapy already created for us, and more specifically, the HtmlXPathSelector (since we're working with an html document right now).
But anyway, we'll stick to the selectors that Scrapy already made for us, and more specifically, the HtmlXPathSelector (since we're working with an HTML document right now).
Let's try some expressions::
@ -63,11 +63,11 @@ Let's try some expressions::
# As you can see, the x method returns an XPathSelectorList, which is actually a list of selectors.
# To extract their data you must use the extract() method, as follows:
In [2]: hxs.x('//title/text()').extract()
Out[2]: [u'My website']
Out[2]: [u'Example website']
# The base url
In [3]: hxs.x('//base/@href').extract()
Out[3]: [u'http://mywebsite.com/']
Out[3]: [u'http://example.com/']
# Image links
In [4]: hxs.x('//a[contains(@href, "image")]/@href').extract()
@ -122,7 +122,8 @@ You can apply an x() call to any node you have, which means that you can join di
There are some things to keep in mind here:
1. | x() calls always return an XPathSelectorList, which is basically a list of selectors, with the extra ability of applying XPath or Regexp to each of its items and returning a new list.
1. | x() calls always return an XPathSelectorList, which is basically a list of selectors, with the extra ability of applying XPath or Regexp to each of its items and
returning a new list.
| That's why you can concatenate x() calls, because they always return XPathSelectorLists, and you can always reapply that method over them.
2. x() calls are relative to the node your standing on, so selector.x('body/div[@id="mydiv"]') equals selector.x('body').x('div[@id="mydiv"]').
3. The extract() method *always* returns a list, even if it contains only one element. Don't forget that.
@ -130,5 +131,5 @@ There are some things to keep in mind here:
| You may also have noticed that I've used another method up there; the re() method.
| This one is very useful when the data extracted by XPath is not enough and you *have to* (remember to not abuse of regexp) make an extra parsing of the information you've got.
| In this cases, you just apply the re() method over any XPathSelector/XPathSelectorList you have with a compiled regexp pattern as the only argument, or a string with the pattern to be compiled.
| Remember that the re() method *always* returns an already extracted list, which means that you can't go back to a node from the result of a re() call (which is actually pretty obvious).
| Remember that the re() method *always* returns an already extracted list, which means that you can't go back to a node from the result of a re() call.