diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 50d571309..c4165e238 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -129,7 +129,6 @@ For more information about XPath see the `XPath reference`_. Finally, here's the spider code:: - import scrapy from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor @@ -141,12 +140,11 @@ Finally, here's the spider code:: rules = [Rule(SgmlLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): - sel = scrapy.Selector(response) torrent = TorrentItem() torrent['url'] = response.url - torrent['name'] = sel.xpath("//h1/text()").extract() - torrent['description'] = sel.xpath("//div[@id='description']").extract() - torrent['size'] = sel.xpath("//div[@id='info-left']/p[2]/text()[2]").extract() + torrent['name'] = response.xpath("//h1/text()").extract() + torrent['description'] = response.xpath("//div[@id='description']").extract() + torrent['size'] = response.xpath("//div[@id='info-left']/p[2]/text()[2]").extract() return torrent The ``TorrentItem`` class is :ref:`defined above `. diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index 8d1693b28..64b9dca62 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -175,9 +175,9 @@ Scrapy creates :class:`scrapy.Request ` objects for each URL in the ``start_urls`` attribute of the Spider, and assigns them the ``parse`` method of the spider as their callback function. -These Requests are scheduled, then executed, and -:class:`scrapy.http.Response` objects are returned and then fed back to the -spider, through the :meth:`~scrapy.spider.Spider.parse` method. +These Requests are scheduled, then executed, and :class:`scrapy.http.Response` +objects are returned and then fed back to the spider, through the +:meth:`~scrapy.spider.Spider.parse` method. Extracting Items ---------------- @@ -210,9 +210,9 @@ These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much more powerful. To learn more about XPath we recommend `this XPath tutorial `_. -For working with XPaths, Scrapy provides a :class:`~scrapy.selector.Selector` -class, which is instantiated with a :class:`~scrapy.http.HtmlResponse` or -:class:`~scrapy.http.XmlResponse` object as first argument. +For working with XPaths, Scrapy provides :class:`~scrapy.selector.Selector` +class and convenient shortcuts to avoid instantiating selectors yourself +everytime you need to select something from a response. You can see selectors as objects that represent nodes in the document structure. So, the first instantiated selectors are associated with the root @@ -262,7 +262,6 @@ This is what the shell looks like:: [s] item {} [s] request [s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> - [s] sel \r\n\r\n>> from scrapy.selector import Selector + >>> from scrapy.http import HtmlResponse - class MySpider(scrapy.Spider): - # ... - def parse(self, response): - sel = scrapy.Selector(response) - # Using XPath query - print sel.xpath('//p') - # Using CSS query - print sel.css('p') - # Nesting queries - print sel.xpath('//div[@foo="bar"]').css('span#bold') +Constructing from text:: + >>> body = 'good' + >>> Selector(text=body).xpath('//span/text()').extract() + [u'good'] + +Constructing from response:: + + >>> response = HtmlResponse(url='http://example.com', body=body) + >>> Selector(response=response).xpath('//span/text()').extract() + [u'good'] + +For convenience, response objects exposes a selector on `.selector` attribute, +it's totally OK to use this shortcut when possible:: + + >>> response.selector.xpath('//span/text()').extract() + [u'good'] + Using selectors --------------- @@ -92,66 +101,73 @@ First, let's open the shell:: scrapy shell http://doc.scrapy.org/en/latest/_static/selectors-sample1.html -Then, after the shell loads, you'll have a selector already instantiated and -ready to use in ``sel`` shell variable. +Then, after the shell loads, you'll have the response available as ``response`` +shell variable, and its attached selector in ``response.selector`` attribute. Since we're dealing with HTML, the selector will automatically use an HTML parser. .. highlight:: python So, by looking at the :ref:`HTML code ` of that -page, let's construct an XPath (using an HTML selector) for selecting the text -inside the title tag:: +page, let's construct an XPath for selecting the text inside the title tag:: - >>> sel.xpath('//title/text()') + >>> response.selector.xpath('//title/text()') [] -As you can see, the ``.xpath()`` method returns an +Querying responses using XPath and CSS is so common that responses includes two +convenient shortcuts: ``response.xpath()`` and ``response.css()``:: + + >>> response.xpath('//title/text()') + [] + >>> response.css('title::text') + [] + +As you can see, the ``.xpath()`` and ``.css()`` methods returns an :class:`~scrapy.selector.SelectorList` instance, which is a list of new selectors. This API can be used quickly for extracting nested data. To actually extract the textual data, you must call the selector ``.extract()`` method, as follows:: - >>> sel.xpath('//title/text()').extract() + >>> response.xpath('//title/text()').extract() [u'Example website'] Notice that CSS selectors can select text or attribute nodes using CSS3 pseudo-elements:: - >>> sel.css('title::text').extract() + >>> response.css('title::text').extract() [u'Example website'] Now we're going to get the base URL and some image links:: - >>> sel.xpath('//base/@href').extract() + >>> response.xpath('//base/@href').extract() [u'http://example.com/'] - >>> sel.css('base::attr(href)').extract() + >>> response.css('base::attr(href)').extract() [u'http://example.com/'] - >>> sel.xpath('//a[contains(@href, "image")]/@href').extract() + >>> response.xpath('//a[contains(@href, "image")]/@href').extract() [u'image1.html', u'image2.html', u'image3.html', u'image4.html', u'image5.html'] - >>> sel.css('a[href*=image]::attr(href)').extract() + >>> response.css('a[href*=image]::attr(href)').extract() [u'image1.html', u'image2.html', u'image3.html', u'image4.html', u'image5.html'] - >>> sel.xpath('//a[contains(@href, "image")]/img/@src').extract() + >>> response.xpath('//a[contains(@href, "image")]/img/@src').extract() [u'image1_thumb.jpg', u'image2_thumb.jpg', u'image3_thumb.jpg', u'image4_thumb.jpg', u'image5_thumb.jpg'] - >>> sel.css('a[href*=image] img::attr(src)').extract() + >>> response.css('a[href*=image] img::attr(src)').extract() [u'image1_thumb.jpg', u'image2_thumb.jpg', u'image3_thumb.jpg', @@ -167,7 +183,7 @@ The selection methods (``.xpath()`` or ``.css()``) returns a list of selectors of the same type, so you can call the selection methods for those selectors too. Here's an example:: - >>> links = sel.xpath('//a[contains(@href, "image")]') + >>> links = response.xpath('//a[contains(@href, "image")]') >>> links.extract() [u'Name: My image 1
', u'Name: My image 2
', @@ -196,7 +212,7 @@ can't construct nested ``.re()`` calls. Here's an example used to extract images names from the :ref:`HTML code ` above:: - >>> sel.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)') + >>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)') [u'My image 1', u'My image 2', u'My image 3', @@ -215,7 +231,7 @@ with ``/``, that XPath will be absolute to the document and not relative to the For example, suppose you want to extract all ``

`` elements inside ``

`` elements. First, you would get all ``
`` elements:: - >>> divs = sel.xpath('//div') + >>> divs = response.xpath('//div') At first, you may be tempted to use the following approach, which is wrong, as it actually extracts all ``

`` elements from the document, not only those @@ -429,6 +445,10 @@ Built-in Selectors reference ``query`` is a string containing the XPATH query to apply. + .. note:: + + For convenience this method can be called as ``response.xpath()`` + .. method:: css(query) Apply the given CSS selector and return a :class:`SelectorList` instance. @@ -438,6 +458,10 @@ Built-in Selectors reference In the background, CSS queries are translated into XPath queries using `cssselect`_ library and run ``.xpath()`` method. + .. note:: + + For convenience this method can be called as ``response.css()`` + .. method:: extract() Serialize and return the matched nodes as a list of unicode strings. @@ -570,14 +594,14 @@ First, we open the shell with the url we want to scrape:: Once in the shell we can try selecting all ```` objects and see that it doesn't work (because the Atom XML namespace is obfuscating those nodes):: - >>> sel.xpath("//link") + >>> response.xpath("//link") [] But once we call the :meth:`Selector.remove_namespaces` method, all nodes can be accessed directly by their names:: - >>> sel.remove_namespaces() - >>> sel.xpath("//link") + >>> response.selector.remove_namespaces() + >>> response.xpath("//link") [, ... diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 929e21e10..c709f1d39 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -231,11 +231,10 @@ Another example returning multiple Requests and Items from a single callback:: ] def parse(self, response): - sel = scrapy.Selector(response) - for h3 in sel.xpath('//h3').extract(): + for h3 in response.xpath('//h3').extract(): yield MyItem(title=h3) - for url in sel.xpath('//a/@href').extract(): + for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse) .. module:: scrapy.contrib.spiders @@ -332,12 +331,10 @@ Let's now take a look at an example CrawlSpider with rules:: def parse_item(self, response): self.log('Hi, this is an item page! %s' % response.url) - - sel = scrapy.Selector(response) item = scrapy.Item() - item['id'] = sel.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') - item['name'] = sel.xpath('//td[@id="item_name"]/text()').extract() - item['description'] = sel.xpath('//td[@id="item_description"]/text()').extract() + item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)') + item['name'] = response.xpath('//td[@id="item_name"]/text()').extract() + item['description'] = response.xpath('//td[@id="item_description"]/text()').extract() return item