`` element, such as::
+.. _XPath: https://www.w3.org/TR/xpath
+.. _CSS: https://www.w3.org/TR/selectors
+
+Extracting quotes and authors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now that you know a bit about selection and extraction, let's complete our
+spider by writing the code to extract the quotes from the web page.
+
+Each quote in http://quotes.toscrape.com is represented by HTML elements that look
+like this:
+
+.. code-block:: html
+Let's open up scrapy shell and play a bit to find out how to extract the data
+we want::
-So we can select each ``
`` element belonging to the site's
-list with this code::
+ $ scrapy shell 'http://quotes.toscrape.com'
- response.xpath('//div[@class="quote"]')
+We get a list of selectors for the quote HTML elements with::
-From the quote elements, we can select the texts with::
+ >>> response.css("div.quote")
- response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
+Each of the selectors returned by the query above allows us to run further
+queries over their sub-elements. Let's assign the first selector to a
+variable, so that we can run our CSS selectors directly on a particular quote::
-The authors::
+ >>> quote = response.css("div.quote")[0]
- response.xpath('//div[@class="quote"]/span/small/text()').extract()
+Now, let's extract ``title``, ``author`` and the ``tags`` from that quote
+using the ``quote`` object we just created::
-As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
-concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
-that property here, so::
+ >>> title = quote.css("span.text::text").extract_first()
+ >>> title
+ '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
+ >>> author = quote.css("small.author::text").extract_first()
+ >>> author
+ 'Albert Einstein'
- for quote in response.xpath('//div[@class="quote"]'):
- text = quote.xpath('span[@class="text"]/text()').extract()
- author = quote.xpath('span/small/text()').extract()
- print('{}: {}'.format(author, text))
+Given that the tags are a list of strings, we can use the ``.extract()`` method
+to get all of them::
-.. note::
+ >>> tags = quote.css("div.tags a.tag::text").extract()
+ >>> tags
+ ['change', 'deep-thoughts', 'thinking', 'world']
- For a more detailed description of using nested selectors, see
- :ref:`topics-selectors-nesting-selectors` and
- :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors`
- documentation
+Having figured out how to extract each bit, we can now iterate over all the
+quotes elements and put them together into a Python dictionary::
-Let's add this code to our spider::
+ >>> for quote in response.css("div.quote"):
+ ... text = quote.css("span.text::text").extract_first()
+ ... author = quote.css("small.author::text").extract_first()
+ ... tags = quote.css("div.tags a.tag::text").extract()
+ ... print(dict(text=text, author=author, tags=tags))
+ {'tags': ['change', 'deep-thoughts', 'thinking', 'world'], 'author': 'Albert Einstein', 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}
+ {'tags': ['abilities', 'choices'], 'author': 'J.K. Rowling', 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”'}
+ ... a few more of these, omitted for brevity
+ >>>
+
+Extracting data in our spider
+------------------------------
+
+Let's get back to our spider. Until now, it doesn't extract any data in
+particular, just saves the whole HTML page to a local file. Let's integrate the
+extraction logic above into our spider.
+
+A Scrapy spider typically generates many dictionaries containing the data
+extracted from the page. To do that, we use the ``yield`` Python keyword
+in the callback, as you can see below::
import scrapy
@@ -379,78 +416,96 @@ Let's add this code to our spider::
]
def parse(self, response):
- for quote in response.xpath('//div[@class="quote"]'):
- text = quote.xpath('span[@class="text"]/text()').extract_first()
- author = quote.xpath('span/small/text()').extract_first()
- print(u'{}: {}'.format(author, text))
+ for quote in response.css('div.quote'):
+ yield {
+ 'text': quote.css('span.text::text').extract_first(),
+ 'author': quote.css('span small::text').extract_first(),
+ 'tags': quote.css('div.tags a.tag::text').extract(),
+ }
-Note how we've changed to use the method ``.extract_first()``, which extracts
-the first element from a selector list returned by ``.xpath()``.
+If you run this spider, it will output the extracted data with the log::
-Now try crawling quotes.toscrape.com again and you'll see sites being printed
-in your output. Run::
-
- scrapy crawl quotes
-
-Using our item
---------------
-
-:class:`~scrapy.item.Item` objects are custom Python dicts; you can access the
-values of their fields (attributes of the class we defined earlier) using the
-standard dict syntax like::
-
- >>> from tutorial.items import QuoteItem
- >>> item = QuoteItem()
- >>> item['text'] = 'Some random quote'
- >>> item['text']
- 'Some random quote'
-
-So, in order to return the data we've scraped so far, the final code for our
-Spider would be like this::
-
- import scrapy
- from tutorial.items import QuoteItem
+ 2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
+ {'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
+ 2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
+ {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}
- class QuotesSpider(scrapy.Spider):
- name = "quotes"
- start_urls = [
- 'http://quotes.toscrape.com/page/1/',
- 'http://quotes.toscrape.com/page/2/',
- ]
+.. _storing-data:
- def parse(self, response):
- for quote in response.xpath('//div[@class="quote"]'):
- item = QuoteItem()
- item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
- item['author'] = quote.xpath('span/small/text()').extract_first()
- yield item
+Storing the scraped data
+========================
+The simplest way to store the scraped data is by using :ref:`Feed exports
+
`, with the following command::
-Now crawling quotes.toscrape.com yields ``QuoteItem`` objects::
+ scrapy crawl quotes -o quotes.json
- 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
- {'author': 'Oscar Wilde',
- 'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
- 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
- {'author': 'Mark Twain',
- 'text': '“The man who does not read has no advantage over the man who cannot read.”'}
+That will generate an ``quotes.json`` file containing all scraped items,
+serialized in `JSON`_.
+
+For historic reasons, Scrapy appends to a given file instead of overwriting
+its contents. If you run this command twice without removing the file
+before the second time, you'll end up with a broken JSON file.
+
+You can also used other formats, like `JSON Lines`_::
+
+ scrapy crawl quotes -o quotes.jl
+
+The `JSON Lines`_ format is useful because it's stream-like, you can easily
+append new records to it. It doesn't have the same problem of JSON when you run
+twice. Also, as each record is a separate line, you can process big files
+without having to fit everything in memory, there are tools like `JQ`_ to help
+doing that at the command-line.
+
+In small projects (like the one in this tutorial), that should be enough.
+However, if you want to perform more complex things with the scraped items, you
+can write an :ref:`Item Pipeline `. A placeholder file
+for Item Pipelines has been set up for you when the project is created, in
+``tutorial/pipelines.py``. Though you don't need to implement any item
+pipelines if you just want to store the scraped items.
+
+.. _JSON Lines: http://jsonlines.org
+.. _JQ: https://stedolan.github.io/jq
Following links
===============
Let's say, instead of just scraping the stuff from the first two pages
-from quotes.toscrape.com, you want quotes from all the pages in the website.
+from http://quotes.toscrape.com, you want quotes from all the pages in the website.
-Now that you know how to extract data from a page, why not extract the
-pagination links in each page, follow them and then extract the data you
-want for all of them?
+Now that you know how to extract data from pages, let's see how to follow links
+from them.
-Here is a modification to our spider that does just that::
+First thing is to extract the link to the page we want to follow. Examining
+our page, we can see there is a link to the next page with the following
+markup:
+
+.. code-block:: html
+
+
+
+We can try extracting it in the shell::
+
+ >>> response.css('li.next a').extract_first()
+ 'Next →'
+
+This gets the anchor element, but we want the attribute ``href``. For that,
+Scrapy supports a CSS extension that let's you select the attribute contents,
+like this::
+
+ >>> response.css('li.next a::attr(href)').extract_first()
+ '/page/2/'
+
+Let's see now our spider modified to recursively follow the link to the next
+page, extracting data from it::
import scrapy
- from tutorial.items import QuoteItem
class QuotesSpider(scrapy.Spider):
@@ -460,19 +515,25 @@ Here is a modification to our spider that does just that::
]
def parse(self, response):
- for quote in response.xpath('//div[@class="quote"]'):
- item = QuoteItem()
- item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
- item['author'] = quote.xpath('span/small/text()').extract_first()
- yield item
- next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
- if next_page:
+ for quote in response.css('div.quote'):
+ yield {
+ 'text': quote.css('span.text::text').extract_first(),
+ 'author': quote.css('span small::text').extract_first(),
+ 'tags': quote.css('div.tags a.tag::text').extract(),
+ }
+
+ next_page = response.css('li.next a::attr(href)').extract_first()
+ if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page, callback=self.parse)
-Now after extracting an item the `parse()` method looks for the link to the next page,
-builds a full absolute URL using the `response.urljoin` method (since the links can
-be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.
+
+Now, after extracting the data, the ``parse()`` method looks for the link to
+the next page, builds a full absolute URL using the
+:meth:`~scrapy.http.Response.urljoin` method (since the links can be
+relative) and yields a new request to the next page, registering itself as
+callback to handle the data extraction for the next page and to keep the
+crawling going through all the pages.
What you see here is Scrapy's mechanism of following links: when you yield
a Request in a callback method, Scrapy will schedule that request to be sent
@@ -486,34 +547,116 @@ In our example, it creates a sort of loop, following all the links to the next p
until it doesn't find one -- handy for crawling blogs, forums and other sites with
pagination.
-Another common pattern is to build an item with data from more than one page,
+More examples and patterns
+--------------------------
+
+Here is another spider that illustrates callbacks and following links,
+this time for scraping author information::
+
+
+ import scrapy
+
+
+ class AuthorSpider(scrapy.Spider):
+ name = 'author'
+
+ start_urls = ['http://quotes.toscrape.com/']
+
+ def parse(self, response):
+ # follow links to author pages
+ for href in response.css('.author a::attr(href)').extract():
+ yield scrapy.Request(response.urljoin(href),
+ callback=self.parse_author)
+
+ # follow pagination links
+ next_page = response.css('li.next a::attr(href)').extract_first()
+ if next_page is not None:
+ next_page = response.urljoin(next_page)
+ yield scrapy.Request(next_page, callback=self.parse)
+
+ def parse_author(self, response):
+ def extract_with_css(query):
+ return response.css(query).extract_first().strip()
+
+ yield {
+ 'name': extract_with_css('h3.author-title::text'),
+ 'birthdate': extract_with_css('.author-born-date::text'),
+ 'bio': extract_with_css('.author-description::text'),
+ }
+
+This spider will start from the main page, it will follow all the links to the
+authors pages calling the ``parse_author`` callback for each of them, and also
+the pagination links with the ``parse`` callback as we saw before.
+
+The ``parse_author`` callback defines a helper function to extract and cleanup the
+data from a CSS query and yields the Python dict with the author data.
+
+Another interesting thing this spider demonstrates is that, even if there are
+many quotes from the same author, we don't need to worry about visiting the
+same author page multiple times. By default, Scrapy filters out duplicated
+requests to URLs already visited, avoiding the problem of hitting servers too
+much because of a programming mistake. This can be configured by the setting
+:setting:`DUPEFILTER_CLASS`.
+
+Hopefully by now you have a good understanding of how to use the mechanism
+of following links and callbacks with Scrapy.
+
+As yet another example spider that leverages the mechanism of following links,
+check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic
+spider that implements a small rules engine that you can use to write your
+crawlers on top of it.
+
+Also, a common pattern is to build an item with data from more than one page,
using a :ref:`trick to pass additional data to the callbacks
`.
-.. note::
- As an example spider that leverages this mechanism, check out the
- :class:`~scrapy.spiders.CrawlSpider` class for a generic spider
- that implements a small rules engine that you can use to write your
- crawlers on top of it.
+Using spider arguments
+======================
-Storing the scraped data
-========================
+You can provide command line arguments to your spiders by using the ``-a``
+option when running them::
-The simplest way to store the scraped data is by using :ref:`Feed exports
-`, with the following command::
+ scrapy crawl quotes -o quotes-humor.json -a tag=humor
- scrapy crawl quotes -o items.json
+These arguments are passed to the Spider's ``__init__`` method and become
+spider attributes by default.
-That will generate an ``items.json`` file containing all scraped items,
-serialized in `JSON`_.
+In this example, the value provided for the ``tag`` argument will be available
+via ``self.tag``. You can use this to make your spider fetch only quotes
+with a specific tag, building the URL based on the argument::
-In small projects (like the one in this tutorial), that should be enough.
-However, if you want to perform more complex things with the scraped items, you
-can write an :ref:`Item Pipeline `. As with Items, a
-placeholder file for Item Pipelines has been set up for you when the project is
-created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
-pipelines if you just want to store the scraped items.
+ import scrapy
+
+
+ class QuotesSpider(scrapy.Spider):
+ name = "quotes"
+
+ def start_requests(self):
+ url = 'http://quotes.toscrape.com/'
+ tag = getattr(self, 'tag', None)
+ if tag is not None:
+ url = url + 'tag/' + tag
+ yield scrapy.Request(url, self.parse)
+
+ def parse(self, response):
+ for quote in response.css('div.quote'):
+ yield {
+ 'text': quote.css('span.text::text').extract_first(),
+ 'author': quote.css('span small a::text').extract_first(),
+ }
+
+ next_page = response.css('li.next a::attr(href)').extract_first()
+ if next_page is not None:
+ next_page = response.urljoin(next_page)
+ yield scrapy.Request(next_page, self.parse)
+
+
+If you pass the ``tag=humor`` argument to this spider, you'll notice that it
+will only visit URLs from the ``humor`` tag, such as
+``http://quotes.toscrape.com/tag/humor``.
+
+You can :ref:`learn more about handling spider arguments here `.
Next steps
==========
@@ -522,9 +665,10 @@ This tutorial covered only the basics of Scrapy, but there's a lot of other
features not mentioned here. Check the :ref:`topics-whatelse` section in
:ref:`intro-overview` chapter for a quick overview of the most important ones.
-Then, we recommend you continue by playing with an example project (see
-:ref:`intro-examples`), and then continue with the section
-:ref:`section-basics`.
+You can continue from the section :ref:`section-basics` to know more about the
+command-line tool, spiders, selectors and other things the tutorial hasn't covered like
+modeling the scraped data. If you prefer to play with an example project, check
+the :ref:`intro-examples` section.
.. _JSON: https://en.wikipedia.org/wiki/JSON
.. _dirbot: https://github.com/scrapy/dirbot