`` element, such as::
“We accept the love we think we deserve.”
by Stephen Chbosky
So we can select each ``
`` element belonging to the site's
list with this code::
response.xpath('//div[@class="quote"]')
From the quote elements, we can select the texts with::
response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
The authors::
response.xpath('//div[@class="quote"]/span/small/text()').extract()
As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
that property here, so::
for quote in response.xpath('//div[@class="quote"]'):
text = quote.xpath('span[@class="text"]/text()').extract()
author = quote.xpath('span/small/text()').extract()
print('{}: {}'.format(author, text))
.. note::
For a more detailed description of using nested selectors, see
:ref:`topics-selectors-nesting-selectors` and
:ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors`
documentation
Let's add this code to our spider::
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/',
]
def parse(self, response):
for quote in response.xpath('//div[@class="quote"]'):
text = quote.xpath('span[@class="text"]/text()').extract_first()
author = quote.xpath('span/small/text()').extract_first()
print(u'{}: {}'.format(author, text))
Note how we've changed to use the method ``.extract_first()``, which extracts
the first element from a selector list returned by ``.xpath()``.
Now try crawling quotes.toscrape.com again and you'll see sites being printed
in your output. Run::
scrapy crawl quotes
Using our item
--------------
:class:`~scrapy.item.Item` objects are custom Python dicts; you can access the
values of their fields (attributes of the class we defined earlier) using the
standard dict syntax like::
>>> from tutorial.items import QuoteItem
>>> item = QuoteItem()
>>> item['text'] = 'Some random quote'
>>> item['title']
'Some random quote'
So, in order to return the data we've scraped so far, the final code for our
Spider would be like this::
import scrapy
from tutorial.items import QuoteItem
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
'http://quotes.toscrape.com/page/2/',
]
def parse(self, response):
for quote in response.xpath('//div[@class="quote"]'):
item = QuoteItem()
item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
item['author'] = quote.xpath('span/small/text()').extract_first()
yield item
Now crawling quotes.toscrape.com yields ``QuoteItem`` objects::
2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'author': 'Oscar Wilde',
'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'author': 'Mark Twain',
'text': '“The man who does not read has no advantage over the man who cannot read.”'}
Following links
===============
Let's say, instead of just scraping the stuff from the first two pages
from quotes.toscrape.com, you want quotes from all the pages in the website.
Now that you know how to extract data from a page, why not extract the
pagination links in each page, follow them and then extract the data you
want for all of them?
Here is a modification to our spider that does just that::
import scrapy
from tutorial.items import QuoteItem
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):
for quote in response.xpath('//div[@class="quote"]'):
item = QuoteItem()
item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
item['author'] = quote.xpath('span/small/text()').extract_first()
yield item
next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
if next_page:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
Now after extracting an item the `parse()` method looks for the link to the next page,
builds a full absolute URL using the `response.urljoin` method (since the links can
be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.
What you see here is Scrapy's mechanism of following links: when you yield
a Request in a callback method, Scrapy will schedule that request to be sent
and register a callback method to be executed when that request finishes.
Using this, you can build complex crawlers that follow links according to rules
you define, and extract different kinds of data depending on the page it's
visiting.
In our example, it creates a sort of loop, following all the links to the next page
until it doesn't find one -- handy for crawling blogs, forums and other sites with
pagination.
Another common pattern is to build an item with data from more than one page,
using a :ref:`trick to pass additional data to the callbacks
`.
.. note::
As an example spider that leverages this mechanism, check out the
:class:`~scrapy.spiders.CrawlSpider` class for a generic spider
that implements a small rules engine that you can use to write your
crawlers on top of it.
Storing the scraped data
========================
The simplest way to store the scraped data is by using :ref:`Feed exports
`, with the following command::
scrapy crawl quotes -o items.json
That will generate an ``items.json`` file containing all scraped items,
serialized in `JSON`_.
In small projects (like the one in this tutorial), that should be enough.
However, if you want to perform more complex things with the scraped items, you
can write an :ref:`Item Pipeline `. As with Items, a
placeholder file for Item Pipelines has been set up for you when the project is
created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
pipelines if you just want to store the scraped items.
Next steps
==========
This tutorial covered only the basics of Scrapy, but there's a lot of other
features not mentioned here. Check the :ref:`topics-whatelse` section in
:ref:`intro-overview` chapter for a quick overview of the most important ones.
Then, we recommend you continue by playing with an example project (see
:ref:`intro-examples`), and then continue with the section
:ref:`section-basics`.
.. _JSON: https://en.wikipedia.org/wiki/JSON
.. _dirbot: https://github.com/scrapy/dirbot