From fee07835f2f9504acc7f0952088ca2ea201027c3 Mon Sep 17 00:00:00 2001 From: Valdir Stumm Junior Date: Mon, 19 Sep 2016 19:19:31 -0300 Subject: [PATCH] Completing the data extraction section --- docs/intro/tutorial.rst | 122 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 112 insertions(+), 10 deletions(-) diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index f6aa6476c..473183be3 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -106,8 +106,8 @@ and defines some attributes and methods: the scraped data as dicts and also finding new URLs to follow and creating new requests (:class:`~scrapy.http.Request`) from them. -How to run your spider ----------------------- +How to run our spider +--------------------- To put our spider to work, go to the project's top level directory and run:: @@ -148,12 +148,14 @@ objects and calls the ``parse`` callback method passing the response as argument. -Simplifying your spider ------------------------ -Instead of defining the :meth:`~scrapy.spiders.Spider.start_requests` method -generating :class:`scrapy.Request ` -objects from URLs, you can just put those URLs in the -:attr:`~scrapy.spiders.Spider.start_urls` attribute:: +A shortcut to the start_requests method +--------------------------------------- +Instead of implementing a :meth:`~scrapy.spiders.Spider.start_requests` method +that generates :class:`scrapy.Request ` objects from URLs, +you can just define a :attr:`~scrapy.spiders.Spider.start_urls` class attribute +with a list of URLs. This list will then be used by the default implementation +of :meth:`~scrapy.spiders.Spider.start_requests` to create the initial requests +for your spider:: import scrapy @@ -174,7 +176,8 @@ objects from URLs, you can just put those URLs in the The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each of the requests for those URLs, even though we haven't explicitely told Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` -is Scrapy's default callback method. +is Scrapy's default callback method that is called for any request that have +been generated with no callback explicitely assigned to handle it. Extracting data @@ -293,7 +296,104 @@ Extraction wrap-up Now that you know a bit about selection and extraction, let's complete our spider by writing the code to extract the quotes from the webpage. -TODO: show how to extract quotes and integrate spider code here. +Each quote in http://quotes.toscrape.com is represented by HTML code that looks +like this:: + +
+ “The world as we have created it is a process of our + thinking. It cannot be changed without changing our thinking.” + + by Albert Einstein + (about) + +
+ Tags: + change + deep-thoughts + thinking + world +
+
+ +Let's open up scrapy shell and play a bit to find out how to extract the data +we want:: + + $ scrapy shell http://quotes.toscrape.com + +We get a list of selectors to the quotes using:: + + >>> response.css("div.quote") + +Each of the selectors returned by the query above allows us to run further +queries over the quotes itselves. Let's assign the first selector to a +variable, so that we can run our CSS selectors directly on a particular quote:: + + >>> quote = response.css("div.quote")[0] + +Now, let's extract ``title``, ``author`` and the ``tags`` from that quote +using the ``quote`` object we just created:: + + >>> title = quote.css("span.text ::text").extract_first() + >>> title + '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”' + >>> author = quote.css("small.author ::text").extract_first() + >>> author + 'Albert Einstein' + +Given that the tags is a list of strings, we can use the ``.extract()`` method +to get all of them:: + + >>> tags = quote.css("div.tags a.tag ::text").extract() + >>> tags + ['change', 'deep-thoughts', 'thinking', 'world'] + +Now, we can iterate over all the quotes in the page and use the CSS selectors +we defined to extract data:: + + >>> for quote in response.css("div.quote"): + ... text = quote.css("span.text ::text").extract_first() + ... author = quote.css("small.author ::text").extract_first() + ... tags = quote.css("div.tags a.tag ::text").extract() + ... print("{} - {} - {}".format(text, author, tags)) + + +Extracting data in our spider +------------------------------ + +Until now, the spider we built doesn't extract any data in particular. I just +saves the whole HTML page to a local file. Now, let's integrate the extraction +logic above in our spider. + +A Scrapy spider typically generates many dictionaries containing the data +extracted from the page. To do that, we use the ``yield`` Python keyword, as +you can see below:: + + import scrapy + + + class QuotesSpider(scrapy.Spider): + name = "quotes" + start_urls = [ + 'http://quotes.toscrape.com/page/1/', + 'http://quotes.toscrape.com/page/2/', + ] + + def parse(self, response): + for quote in response.css('div.quote'): + yield { + 'text': quote.css('span.text::text').extract_first(), + 'author': quote.css('span small::text').extract_first(), + 'tags': quote.css("div.tags a.tag ::text").extract(), + } + +If you run this spider, it will output the extracted data with the log:: + + 2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> + {'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'} + 2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/> + {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"} + +:ref:`Later in the tutorial `, we will see how to save this data to a file. Following links @@ -450,6 +550,8 @@ If you pass the ``tag=humor`` argument to this spider, you'll notice that it will only visit URLs from the ``humor`` tag, such as ``http://quotes.toscrape.com/tag/humor``. +.. _storing-data: + Storing the scraped data ========================