`` element belonging to the site's list with this
-code::
+
+
“We accept the love we think we deserve.”
+
by Stephen Chbosky
+
+
`` element belonging to the site's
+list with this code::
- response.xpath('//ul/li/text()').extract()
+ response.xpath('//div[@class="quote"]')
-The site's titles::
+From the quote elements, we can select the texts with::
- response.xpath('//ul/li/a/text()').extract()
+ response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
-And the site's links::
+The authors::
- response.xpath('//ul/li/a/@href').extract()
+ response.xpath('//div[@class="quote"]/span/small/text()').extract()
As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
that property here, so::
- for sel in response.xpath('//ul/li'):
- title = sel.xpath('a/text()').extract()
- link = sel.xpath('a/@href').extract()
- desc = sel.xpath('text()').extract()
- print title, link, desc
+ for quote in response.xpath('//div[@class="quote"]'):
+ text = quote.xpath('span[@class="text"]/text()').extract()
+ author = quote.xpath('span/small/text()').extract()
+ print('{}: {}'.format(author, text))
.. note::
@@ -362,26 +369,28 @@ that property here, so::
Let's add this code to our spider::
import scrapy
-
- class DmozSpider(scrapy.Spider):
- name = "dmoz"
- allowed_domains = ["dmoz.org"]
- start_urls = [
- "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
- "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
- ]
-
- def parse(self, response):
- for sel in response.xpath('//ul/li'):
- title = sel.xpath('a/text()').extract()
- link = sel.xpath('a/@href').extract()
- desc = sel.xpath('text()').extract()
- print title, link, desc
-Now try crawling dmoz.org again and you'll see sites being printed
+
+ class QuotesSpider(scrapy.Spider):
+ name = "quotes"
+ start_urls = [
+ 'http://quotes.toscrape.com/page/1/',
+ 'http://quotes.toscrape.com/page/2/',
+ ]
+
+ def parse(self, response):
+ for quote in response.xpath('//div[@class="quote"]'):
+ text = quote.xpath('span[@class="text"]/text()').extract_first()
+ author = quote.xpath('span/small/text()').extract_first()
+ print(u'{}: {}'.format(author, text))
+
+Note how we've changed to use the method ``.extract_first()``, which extracts
+the first element from a selector list returned by ``.xpath()``.
+
+Now try crawling quotes.toscrape.com again and you'll see sites being printed
in your output. Run::
- scrapy crawl dmoz
+ scrapy crawl quotes
Using our item
--------------
@@ -389,91 +398,81 @@ Using our item
:class:`~scrapy.item.Item` objects are custom Python dicts; you can access the
values of their fields (attributes of the class we defined earlier) using the
standard dict syntax like::
-
- >>> item = DmozItem()
- >>> item['title'] = 'Example title'
+
+ >>> from tutorial.items import QuoteItem
+ >>> item = QuoteItem()
+ >>> item['text'] = 'Some random quote'
>>> item['title']
- 'Example title'
+ 'Some random quote'
So, in order to return the data we've scraped so far, the final code for our
Spider would be like this::
import scrapy
+ from tutorial.items import QuoteItem
- from tutorial.items import DmozItem
- class DmozSpider(scrapy.Spider):
- name = "dmoz"
- allowed_domains = ["dmoz.org"]
+ class QuotesSpider(scrapy.Spider):
+ name = "quotes"
start_urls = [
- "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
- "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
+ 'http://quotes.toscrape.com/page/1/',
+ 'http://quotes.toscrape.com/page/2/',
]
def parse(self, response):
- for sel in response.xpath('//ul/li'):
- item = DmozItem()
- item['title'] = sel.xpath('a/text()').extract()
- item['link'] = sel.xpath('a/@href').extract()
- item['desc'] = sel.xpath('text()').extract()
+ for quote in response.xpath('//div[@class="quote"]'):
+ item = QuoteItem()
+ item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
+ item['author'] = quote.xpath('span/small/text()').extract_first()
yield item
-.. note:: You can find a fully-functional variant of this spider in the dirbot_
- project available at https://github.com/scrapy/dirbot
-Now crawling dmoz.org yields ``DmozItem`` objects::
+Now crawling quotes.toscrape.com yields ``QuoteItem`` objects::
- [scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
- {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],
- 'link': [u'http://gnosis.cx/TPiP/'],
- 'title': [u'Text Processing in Python']}
- [scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
- {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],
- 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'],
- 'title': [u'XML Processing with Python']}
+ 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
+ {'author': 'Oscar Wilde',
+ 'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
+ 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
+ {'author': 'Mark Twain',
+ 'text': '“The man who does not read has no advantage over the man who cannot read.”'}
Following links
===============
-Let's say, instead of just scraping the stuff in *Books* and *Resources* pages,
-you want everything that is under the `Python directory
-`_.
+Let's say, instead of just scraping the stuff from the first two pages
+from quotes.toscrape.com, you want quotes from all the pages in the website.
-Now that you know how to extract data from a page, why not extract the links
-for the pages you are interested, follow them and then extract the data you
+Now that you know how to extract data from a page, why not extract the
+pagination links in each page, follow them and then extract the data you
want for all of them?
Here is a modification to our spider that does just that::
import scrapy
+ from tutorial.items import QuoteItem
- from tutorial.items import DmozItem
- class DmozSpider(scrapy.Spider):
- name = "dmoz"
- allowed_domains = ["dmoz.org"]
+ class QuotesSpider(scrapy.Spider):
+ name = "quotes"
start_urls = [
- "http://www.dmoz.org/Computers/Programming/Languages/Python/",
+ 'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):
- for href in response.css("ul.directory.dir-col > li > a::attr('href')"):
- url = response.urljoin(href.extract())
- yield scrapy.Request(url, callback=self.parse_dir_contents)
-
- def parse_dir_contents(self, response):
- for sel in response.xpath('//ul/li'):
- item = DmozItem()
- item['title'] = sel.xpath('a/text()').extract()
- item['link'] = sel.xpath('a/@href').extract()
- item['desc'] = sel.xpath('text()').extract()
+ for quote in response.xpath('//div[@class="quote"]'):
+ item = QuoteItem()
+ item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
+ item['author'] = quote.xpath('span/small/text()').extract_first()
yield item
+ next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
+ if next_page:
+ next_page = response.urljoin(next_page)
+ yield scrapy.Request(next_page, callback=self.parse)
-Now the `parse()` method only extracts the interesting links from the page,
+Now after extracting an item the `parse()` method looks for the link to the next page,
builds a full absolute URL using the `response.urljoin` method (since the links can
-be relative) and yields new requests to be sent later, registering as callback
-the method `parse_dir_contents()` that will ultimately scrape the data we want.
+be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.
What you see here is Scrapy's mechanism of following links: when you yield
a Request in a callback method, Scrapy will schedule that request to be sent
@@ -483,25 +482,8 @@ Using this, you can build complex crawlers that follow links according to rules
you define, and extract different kinds of data depending on the page it's
visiting.
-A common pattern is a callback method that extracts some items, looks for a link
-to follow to the next page and then yields a `Request` with the same callback
-for it::
-
- def parse_articles_follow_next_page(self, response):
- for article in response.xpath("//article"):
- item = ArticleItem()
-
- ... extract article data here
-
- yield item
-
- next_page = response.css("ul.navigation > li.next-page > a::attr('href')")
- if next_page:
- url = response.urljoin(next_page[0].extract())
- yield scrapy.Request(url, self.parse_articles_follow_next_page)
-
-This creates a sort of loop, following all the links to the next page until it
-doesn't find one -- handy for crawling blogs, forums and other sites with
+In our example, it creates a sort of loop, following all the links to the next page
+until it doesn't find one -- handy for crawling blogs, forums and other sites with
pagination.
Another common pattern is to build an item with data from more than one page,
@@ -521,7 +503,7 @@ Storing the scraped data
The simplest way to store the scraped data is by using :ref:`Feed exports
`, with the following command::
- scrapy crawl dmoz -o items.json
+ scrapy crawl quotes -o items.json
That will generate an ``items.json`` file containing all scraped items,
serialized in `JSON`_.