From 2427791287d14f14b846c67dc1edff40a1d2b778 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Thu, 15 Sep 2016 17:46:31 -0300
Subject: [PATCH 01/20] tutorial: remove item class definition and present
 start_requests first This changes the tutorial, removing the step of creating
 an item class and also starts by presenting the start_requests method instead
 of start_urls.

---
 docs/intro/tutorial.rst | 139 +++++++++++++---------------------------
 1 file changed, 43 insertions(+), 96 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f802c4e49..0a3361799 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -13,10 +13,9 @@ our example domain to scrape.
 This tutorial will walk you through these tasks:
 
 1. Creating a new Scrapy project
-2. Defining the Items you will extract
-3. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract
+2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract
    :ref:`Items <topics-items>`
-4. Exporting the scraped data using command line
+3. Exporting the scraped data using command line
 
 Scrapy is written in Python_. If you're new to the language you might want to
 start by getting an idea of what the language is like, to get the most out of
@@ -55,34 +54,6 @@ This will create a ``tutorial`` directory with the following contents::
                 __init__.py
 
 
-Defining our Item
-=================
-
-`Items` are containers that will be loaded with the scraped data; they work
-like simple Python dicts. While you can use plain Python dicts with Scrapy,
-`Items` provide additional protection against populating undeclared fields,
-preventing typos. They can also be used with :ref:`Item Loaders
-<topics-loaders>`, a mechanism with helpers to conveniently populate `Items`.
-
-They are declared by creating a :class:`scrapy.Item <scrapy.item.Item>` class and defining
-its attributes as :class:`scrapy.Field <scrapy.item.Field>` objects, much like in an ORM
-(don't worry if you're not familiar with ORMs, you will see that this is an
-easy task).
-
-We begin by modeling the item that we will use to hold the site's data obtained
-from quotes.toscrape.com. As we want to capture the text and author from each of
-the quotes listed there, we define fields for each of these three attributes. To do that, we edit
-``items.py``, found in the ``tutorial`` directory. Our Item class looks like this::
-
-    import scrapy
-
-    class QuoteItem(scrapy.Item):
-        text = scrapy.Field()
-        author = scrapy.Field()
-
-This may seem complicated at first, but defining an item class allows you to use other handy
-components and helpers within Scrapy.
-
 Our first Spider
 ================
 
@@ -93,20 +64,23 @@ They define an initial list of URLs to download, how to follow links, and how
 to parse the contents of pages to extract :ref:`items <topics-items>`.
 
 To create a Spider, you must subclass :class:`scrapy.Spider
-<scrapy.spiders.Spider>` and define some attributes:
+<scrapy.spiders.Spider>` and define some attributes and methods:
 
 * :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
   unique within a project, that is, you can't set the same name for different
   Spiders.
 
-* :attr:`~scrapy.spiders.Spider.start_urls`: a list of URLs where the
-  Spider will begin to crawl from. The first pages downloaded will be those
-  listed here. The subsequent URLs will be generated successively from data
-  contained in the start URLs.
+* :meth:`~scrapy.spiders.Spider.start_requests`: must return a list
+  of requests where the Spider will begin to crawl from.
+  Subsequent requests will be generated successively from these initial requests.
+
+  As alternative to defining this method, you can define a class
+  attribute :attr:`~scrapy.spiders.Spider.start_urls`, which the default
+  implementation of this method will use to create the proper requests.
 
 * :meth:`~scrapy.spiders.Spider.parse`: a method of the spider, which will
   be called with the downloaded :class:`~scrapy.http.Response` object of each
-  start URL. The response is passed to the method as the first and only
+  initial request. The response is passed to the method as the first and only
   argument.
 
   This method is responsible for parsing the response data and extracting
@@ -124,13 +98,16 @@ This is the code for our first Spider; save it in a file named
 
     class QuotesSpider(scrapy.Spider):
         name = "quotes"
-        start_urls = [
-            'http://quotes.toscrape.com/page/1/',
-            'http://quotes.toscrape.com/page/2/',
-        ]
+
+        def start_requests(self):
+            base_url = 'http://quotes.toscrape.com'
+            for path in ['/page/1/', '/page/2/']:
+                yield scrapy.Request(url=base_url + path,
+                                     callback=self.parse)
 
         def parse(self, response):
-            filename = 'quotes-' + response.url.split("/")[-2] + '.html'
+            page = response.url.split("/")[-2]
+            filename = 'quotes-%s.html' % page
             with open(filename, 'wb') as f:
                 f.write(response.body)
 
@@ -171,13 +148,13 @@ URLs, as our ``parse`` method instructs.
 What just happened under the hood?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Scrapy creates :class:`scrapy.Request <scrapy.http.Request>` objects
-for each URL in the ``start_urls`` attribute of the Spider, and assigns
-them the ``parse`` method of the spider as their callback function.
+Scrapy will schedule the :class:`scrapy.Request <scrapy.http.Request>` objects
+returned by the ``start_requests`` method of the Spider, and when receiving
+a response for each one it will instantiate :class:`scrapy.http.Response`
+objects and call the ``parse`` callback method passing the response as argument.
 
-These Requests are scheduled, then executed, and :class:`scrapy.http.Response`
-objects are returned and then fed back to the spider, through the
-:meth:`~scrapy.spiders.Spider.parse` method.
+.. TODO: add here an explanation about how this structure is so command that
+   we can do a short version of the spider w/ start_urls and default callback
 
 Extracting Items
 ----------------
@@ -355,9 +332,13 @@ concatenate further ``.xpath()`` calls to dig deeper into a node. We are going t
 that property here, so::
 
     for quote in response.xpath('//div[@class="quote"]'):
-        text = quote.xpath('span[@class="text"]/text()').extract()
-        author = quote.xpath('span/small/text()').extract()
-        print('{}: {}'.format(author, text))
+        text = quote.xpath('span[@class="text"]/text()').extract_first()
+        author = quote.xpath('span/small/text()').extract_first()
+        print({'text': text, 'author': author})
+
+In the above snippet we've decided to use the method ``.extract_first()``
+instead of ``.extract()``, to extract the content from the first element from a
+selector list returned by ``.xpath()``.
 
 .. note::
 
@@ -366,7 +347,11 @@ that property here, so::
     :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors`
     documentation
 
-Let's add this code to our spider::
+Knowing to use selectors, extracting data from a page is just a matter of
+yield the Python dictionaries from the callback method instead of printing
+them.
+
+Let's add the necessary code to our spider::
 
     import scrapy
 
@@ -380,54 +365,16 @@ Let's add this code to our spider::
 
         def parse(self, response):
             for quote in response.xpath('//div[@class="quote"]'):
-                text = quote.xpath('span[@class="text"]/text()').extract_first()
-                author = quote.xpath('span/small/text()').extract_first()
-                print(u'{}: {}'.format(author, text))
+                yield {
+                    'text': quote.xpath('span[@class="text"]/text()').extract_first(),
+                    'author': quote.xpath('span/small/text()').extract_first(),
+                }
 
-Note how we've changed to use the method ``.extract_first()``, which extracts
-the first element from a selector list returned by ``.xpath()``.
-
-Now try crawling quotes.toscrape.com again and you'll see sites being printed
-in your output. Run::
+Run::
 
     scrapy crawl quotes
 
-Using our item
---------------
-
-:class:`~scrapy.item.Item` objects are custom Python dicts; you can access the
-values of their fields (attributes of the class we defined earlier) using the
-standard dict syntax like::
-    
-    >>> from tutorial.items import QuoteItem
-    >>> item = QuoteItem()
-    >>> item['text'] = 'Some random quote'
-    >>> item['title']
-    'Some random quote'
-
-So, in order to return the data we've scraped so far, the final code for our
-Spider would be like this::
-
-    import scrapy
-    from tutorial.items import QuoteItem
-
-
-    class QuotesSpider(scrapy.Spider):
-        name = "quotes"
-        start_urls = [
-            'http://quotes.toscrape.com/page/1/',
-            'http://quotes.toscrape.com/page/2/',
-        ]
-
-        def parse(self, response):
-            for quote in response.xpath('//div[@class="quote"]'):
-                item = QuoteItem()
-                item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
-                item['author'] = quote.xpath('span/small/text()').extract_first()
-                yield item
-
-
-Now crawling quotes.toscrape.com yields ``QuoteItem`` objects::
+Now crawling quotes.toscrape.com will show dictionary objects::
 
     2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
     {'author': 'Oscar Wilde',

From c508f406892f9d38860fedf1caf8a41fc69bc184 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Thu, 15 Sep 2016 18:05:09 -0300
Subject: [PATCH 02/20] use harcoded URLs, remove item reference on second
 spider

---
 docs/intro/tutorial.rst | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 0a3361799..d160bfc5c 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -100,10 +100,12 @@ This is the code for our first Spider; save it in a file named
         name = "quotes"
 
         def start_requests(self):
-            base_url = 'http://quotes.toscrape.com'
-            for path in ['/page/1/', '/page/2/']:
-                yield scrapy.Request(url=base_url + path,
-                                     callback=self.parse)
+            urls = [
+                'http://quotes.toscrape.com/page/1/',
+                'http://quotes.toscrape.com/page/2/',
+            ]
+            for url in urls:
+                yield scrapy.Request(url=url, callback=self.parse)
 
         def parse(self, response):
             page = response.url.split("/")[-2]
@@ -397,7 +399,6 @@ want for all of them?
 Here is a modification to our spider that does just that::
 
     import scrapy
-    from tutorial.items import QuoteItem
 
 
     class QuotesSpider(scrapy.Spider):
@@ -408,12 +409,13 @@ Here is a modification to our spider that does just that::
 
         def parse(self, response):
             for quote in response.xpath('//div[@class="quote"]'):
-                item = QuoteItem()
-                item['text'] = quote.xpath('span[@class="text"]/text()').extract_first()
-                item['author'] = quote.xpath('span/small/text()').extract_first()
-                yield item
+                yield {
+                    'text': quote.xpath('span[@class="text"]/text()').extract_first(),
+                    'author': quote.xpath('span/small/text()').extract_first(),
+                }
+
             next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
-            if next_page:
+            if next_page is not None:
                 next_page = response.urljoin(next_page)
                 yield scrapy.Request(next_page, callback=self.parse)
 

From 0da497cf7a5063accfabde3acaf2568bfd2b57e4 Mon Sep 17 00:00:00 2001
From: Valdir Stumm Junior <stummjr@gmail.com>
Date: Fri, 16 Sep 2016 11:55:23 -0300
Subject: [PATCH 03/20] updates on the first section (our first spider)

---
 docs/intro/tutorial.rst | 121 +++++++++++++++++++++++-----------------
 1 file changed, 70 insertions(+), 51 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index d160bfc5c..9ab194865 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -7,8 +7,8 @@ Scrapy Tutorial
 In this tutorial, we'll assume that Scrapy is already installed on your system.
 If that's not the case, see :ref:`intro-install`.
 
-We are going to use `quotes.toscrape.com <http://quotes.toscrape.com/>`_ as
-our example domain to scrape.
+We are going to scrape `quotes.toscrape.com <http://quotes.toscrape.com/>`_, a website
+that lists quotes from famous authors.
 
 This tutorial will walk you through these tasks:
 
@@ -57,41 +57,13 @@ This will create a ``tutorial`` directory with the following contents::
 Our first Spider
 ================
 
-Spiders are classes that you define and Scrapy uses to scrape information from a
-domain (or group of domains).
+Spiders are classes that you define and that Scrapy uses to scrape information
+from a website (or group of websites). They define an initial list of URLs to
+download, how to follow links, and how to parse the the downloaded page contents
+to extract :ref:`items <topics-items>`.
 
-They define an initial list of URLs to download, how to follow links, and how
-to parse the contents of pages to extract :ref:`items <topics-items>`.
-
-To create a Spider, you must subclass :class:`scrapy.Spider
-<scrapy.spiders.Spider>` and define some attributes and methods:
-
-* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
-  unique within a project, that is, you can't set the same name for different
-  Spiders.
-
-* :meth:`~scrapy.spiders.Spider.start_requests`: must return a list
-  of requests where the Spider will begin to crawl from.
-  Subsequent requests will be generated successively from these initial requests.
-
-  As alternative to defining this method, you can define a class
-  attribute :attr:`~scrapy.spiders.Spider.start_urls`, which the default
-  implementation of this method will use to create the proper requests.
-
-* :meth:`~scrapy.spiders.Spider.parse`: a method of the spider, which will
-  be called with the downloaded :class:`~scrapy.http.Response` object of each
-  initial request. The response is passed to the method as the first and only
-  argument.
-
-  This method is responsible for parsing the response data and extracting
-  scraped data (as scraped items) and more URLs to follow.
-
-  The :meth:`~scrapy.spiders.Spider.parse` method is in charge of processing
-  the response and returning scraped data (as :class:`~scrapy.item.Item`
-  objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects).
-
-This is the code for our first Spider; save it in a file named
-``quotes_spider.py`` under the ``tutorial/spiders`` directory::
+This is the code for our first Spider. Save it in a file named
+``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::
 
     import scrapy
 
@@ -113,8 +85,29 @@ This is the code for our first Spider; save it in a file named
             with open(filename, 'wb') as f:
                 f.write(response.body)
 
-Crawling
---------
+
+As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>`
+and defines some attributes and methods:
+
+* :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be
+  unique within a project, that is, you can't set the same name for different
+  Spiders.
+
+* :meth:`~scrapy.spiders.Spider.start_requests`: must return a list
+  of requests where the Spider will begin to crawl from.
+  Subsequent requests will be generated successively from these initial requests.
+
+* :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
+  the response downloaded for each of the requests made. The response parameter
+  is an instance of :class:`~scrapy.http.Response` that holds the page content and
+  has further helpful methods to handle it.
+
+  The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
+  the scraped data as items (:class:`~scrapy.item.Item`) and also finding new URLs to
+  follow and creating new requests (:class:`~scrapy.http.Request`) from them.
+
+How to run your spider
+----------------------
 
 To put our spider to work, go to the project's top level directory and run::
 
@@ -138,25 +131,51 @@ similar to this::
     2016-09-01 16:51:29 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
     2016-09-01 16:51:29 [scrapy] INFO: Closing spider (finished)
 
-.. note::
-    At the end you can see a log line for each URL defined in ``start_urls``.
-    Because these URLs are the starting ones, they have no referrers, which is
-    shown at the end of the log line, where it says ``(referer: None)``.
+Now, check the files in the current directory. You should notice that two new
+files have been created: *quotes-1.html* and *quotes-2.html*, with the content
+for the respective URLs, as our ``parse`` method instructs.
+
+.. note:: If you are wondering why we haven't parsed the HTML yet, hold
+  on, we will cover that soon.
 
-Now, check the files in the current directory. You should notice two new files
-have been created: *quotes-1.html* and *quotes-2.html*, with the content for the respective
-URLs, as our ``parse`` method instructs.
 
 What just happened under the hood?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
+returned by the ``start_requests`` method of the Spider. Upon receiving
+a response for each one, it instantiates :class:`scrapy.http.Response`
+objects and calls the ``parse`` callback method passing the response as
+argument.
 
-Scrapy will schedule the :class:`scrapy.Request <scrapy.http.Request>` objects
-returned by the ``start_requests`` method of the Spider, and when receiving
-a response for each one it will instantiate :class:`scrapy.http.Response`
-objects and call the ``parse`` callback method passing the response as argument.
 
-.. TODO: add here an explanation about how this structure is so command that
-   we can do a short version of the spider w/ start_urls and default callback
+Simplifying your spider
+-----------------------
+Instead of defining the :meth:`~scrapy.spiders.Spider.start_requests` method
+generating :class:`scrapy.Request <scrapy.http.Request>`
+objects from URLs, you can just put those URLs in the 
+:attr:`~scrapy.spiders.Spider.start_urls` attribute::
+
+    import scrapy
+
+
+    class QuotesSpider(scrapy.Spider):
+        name = "quotes"
+        start_urls = [
+          'http://quotes.toscrape.com/page/1/',
+          'http://quotes.toscrape.com/page/2/',
+        ]
+
+        def parse(self, response):
+            page = response.url.split("/")[-2]
+            filename = 'quotes-%s.html' % page
+            with open(filename, 'wb') as f:
+                f.write(response.body)
+
+The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle
+each of the requests for those URLs, even though we haven't explicitely told
+Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse`
+is Scrapy's default callback method.
+
 
 Extracting Items
 ----------------

From 0cd9dfcc85433b5c09049630fe87f08e9ebd36da Mon Sep 17 00:00:00 2001
From: Valdir Stumm Junior <stummjr@gmail.com>
Date: Fri, 16 Sep 2016 15:21:49 -0300
Subject: [PATCH 04/20] small fixes on tutorial

---
 docs/intro/tutorial.rst | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 9ab194865..2bbf71573 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -13,8 +13,7 @@ that lists quotes from famous authors.
 This tutorial will walk you through these tasks:
 
 1. Creating a new Scrapy project
-2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract
-   :ref:`Items <topics-items>`
+2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract data
 3. Exporting the scraped data using command line
 
 Scrapy is written in Python_. If you're new to the language you might want to
@@ -58,9 +57,9 @@ Our first Spider
 ================
 
 Spiders are classes that you define and that Scrapy uses to scrape information
-from a website (or group of websites). They define an initial list of URLs to
-download, how to follow links, and how to parse the the downloaded page contents
-to extract :ref:`items <topics-items>`.
+from a website (or group of websites). They define the initial requests to make,
+how to follow links in the pages, and how to parse the downloaded page content
+to extract data.
 
 This is the code for our first Spider. Save it in a file named
 ``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::
@@ -103,7 +102,7 @@ and defines some attributes and methods:
   has further helpful methods to handle it.
 
   The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
-  the scraped data as items (:class:`~scrapy.item.Item`) and also finding new URLs to
+  the scraped data as dicts and also finding new URLs to
   follow and creating new requests (:class:`~scrapy.http.Request`) from them.
 
 How to run your spider

From b2a5cddbb01e3970ce1e38e821a76c7d10520845 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Fri, 16 Sep 2016 15:44:39 -0300
Subject: [PATCH 05/20] tutorial: update section about following links, expand
 examples adding an AuthorSpider to demonstrate further a different crawling
 arrangement.

---
 docs/intro/tutorial.rst | 78 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 65 insertions(+), 13 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 2bbf71573..f3b933d28 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -408,13 +408,13 @@ Following links
 ===============
 
 Let's say, instead of just scraping the stuff from the first two pages
-from quotes.toscrape.com, you want quotes from all the pages in the website.
+from http://quotes.toscrape.com, you want quotes from all the pages in the website.
 
-Now that you know how to extract data from a page, why not extract the
-pagination links in each page, follow them and then extract the data you
-want for all of them?
+Now that you know how to extract data from pages, let's see how to follow links
+from them.
 
-Here is a modification to our spider that does just that::
+Here is a modification of our spider that recursively follows the link to the next
+page, extracting data from it::
 
     import scrapy
 
@@ -426,18 +426,19 @@ Here is a modification to our spider that does just that::
         ]
 
         def parse(self, response):
-            for quote in response.xpath('//div[@class="quote"]'):
+            for quote in response.css('div.quote'):
                 yield {
-                    'text': quote.xpath('span[@class="text"]/text()').extract_first(),
-                    'author': quote.xpath('span/small/text()').extract_first(),
+                    'text': quote.css('span.text::text').extract_first(),
+                    'author': quote.css('span small::text').extract_first(),
                 }
 
-            next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
+            next_page = response.css('li.next a::attr("href")').extract_first()
             if next_page is not None:
                 next_page = response.urljoin(next_page)
                 yield scrapy.Request(next_page, callback=self.parse)
 
-Now after extracting an item the `parse()` method looks for the link to the next page, 
+
+Now, after extracting the data, the `parse()` method looks for the link to the next page,
 builds a full absolute URL using the `response.urljoin` method (since the links can
 be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.
 
@@ -457,13 +458,64 @@ Another common pattern is to build an item with data from more than one page,
 using a :ref:`trick to pass additional data to the callbacks
 <topics-request-response-ref-request-callback-arguments>`.
 
+Another example: scraping authors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is another spider that illustrates callbacks and following links,
+this time for scraping author information::
+
+
+    import scrapy
+
+
+    class AuthorSpider(scrapy.Spider):
+        name = 'author'
+
+        start_urls = ['http://quotes.toscrape.com/']
+
+        def parse(self, response):
+            # follow links to author pages
+            for href in response.css('.author a::attr("href")').extract():
+                yield scrapy.Request(response.urljoin(href),
+                                     callback=self.parse_author)
+
+            # follow pagination links
+            next_page = response.css('li.next a::attr("href")').extract_first()
+            if next_page is not None:
+                next_page = response.urljoin(next_page)
+                yield scrapy.Request(next_page, callback=self.parse)
+
+        def parse_author(self, response):
+            def extract_with_css(query):
+                return response.css(query).extract_first().strip()
+
+            yield {
+                'name': extract_with_css('h3.author-title::text'),
+                'birthdate': extract_with_css('.author-born-date::text'),
+                'bio': extract_with_css('.author-description::text'),
+            }
+
+This spider will start from the main page, it will follow all the links to the
+authors pages calling the ``parse_author`` callback for each of them, and also
+the paginations links too with the ``parse`` callback as we saw before.
+
+The ``parse_author`` callback defines a helper function to extract and cleanup the
+data from a CSS query and yields the Python dict with the author data.
+
+Another interesting this spider demonstrates is that, even if there are many
+quotes from the same author, we don't need to worry about visiting the same
+page multiple times because Scrapy by default filters out duplicated requests
+to URLs already visited, avoiding the problem of hitting servers too much
+because of a programming mistake. This can be configured by the setting
+:setting:`DUPEFILTER_CLASS`.
 
 .. note::
-    As an example spider that leverages this mechanism, check out the
-    :class:`~scrapy.spiders.CrawlSpider` class for a generic spider
-    that implements a small rules engine that you can use to write your
+    As another example spider that leverages the mechanism of following links,
+    check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic
+    spider that implements a small rules engine that you can use to write your
     crawlers on top of it.
 
+
 Storing the scraped data
 ========================
 

From 21de617c77ca49d7ab09f8721676c449d793cdb7 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Fri, 16 Sep 2016 15:55:14 -0300
Subject: [PATCH 06/20] mention that spiders need to subclass scrapy.Spider

---
 docs/intro/tutorial.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f3b933d28..62304de2c 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -57,9 +57,10 @@ Our first Spider
 ================
 
 Spiders are classes that you define and that Scrapy uses to scrape information
-from a website (or group of websites). They define the initial requests to make,
-how to follow links in the pages, and how to parse the downloaded page content
-to extract data.
+from a website (or group of websites). They must subclass
+:class:`scrapy.Spider` and define the initial requests to make, how to follow
+links in the pages, and how to parse the downloaded page content to extract
+data.
 
 This is the code for our first Spider. Save it in a file named
 ``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::

From 147e75602d52d66ea0d3f385dd34f9cedcab883e Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Fri, 16 Sep 2016 16:47:24 -0300
Subject: [PATCH 07/20] update after review comments (thanks @stummjr)

---
 docs/intro/tutorial.rst | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 62304de2c..a3a0ab390 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -439,9 +439,11 @@ page, extracting data from it::
                 yield scrapy.Request(next_page, callback=self.parse)
 
 
-Now, after extracting the data, the `parse()` method looks for the link to the next page,
-builds a full absolute URL using the `response.urljoin` method (since the links can
-be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.
+Now, after extracting the data, the ``parse()`` method looks for the link to
+the next page, builds a full absolute URL using the ``response.urljoin`` method
+(since the links can be relative) and yields a new request to the next page,
+registering itself as callback to handle the data extraction for the next page
+and to keep the crawling going through all the pages.
 
 What you see here is Scrapy's mechanism of following links: when you yield
 a Request in a callback method, Scrapy will schedule that request to be sent
@@ -498,16 +500,16 @@ this time for scraping author information::
 
 This spider will start from the main page, it will follow all the links to the
 authors pages calling the ``parse_author`` callback for each of them, and also
-the paginations links too with the ``parse`` callback as we saw before.
+the pagination links too with the ``parse`` callback as we saw before.
 
 The ``parse_author`` callback defines a helper function to extract and cleanup the
 data from a CSS query and yields the Python dict with the author data.
 
-Another interesting this spider demonstrates is that, even if there are many
-quotes from the same author, we don't need to worry about visiting the same
-page multiple times because Scrapy by default filters out duplicated requests
-to URLs already visited, avoiding the problem of hitting servers too much
-because of a programming mistake. This can be configured by the setting
+Another interesting thing this spider demonstrates is that, even if there are
+many quotes from the same author, we don't need to worry about visiting the
+same author page multiple times. By default, Scrapy filters out duplicated
+requests to URLs already visited, avoiding the problem of hitting servers too
+much because of a programming mistake. This can be configured by the setting
 :setting:`DUPEFILTER_CLASS`.
 
 .. note::

From 31545a9f84785edf309eb8a1c7238f46d024cdc2 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Fri, 16 Sep 2016 17:13:24 -0300
Subject: [PATCH 08/20] tutorial: updating extracting data section to introduce
 CSS and XPath equally

---
 docs/intro/tutorial.rst | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index a3a0ab390..34d33a9b3 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -177,8 +177,8 @@ Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse`
 is Scrapy's default callback method.
 
 
-Extracting Items
-----------------
+Extracting data
+---------------
 
 Introduction to Selectors
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -191,25 +191,31 @@ mechanisms see the :ref:`Selectors documentation <topics-selectors>`.
 .. _XPath: https://www.w3.org/TR/xpath
 .. _CSS: https://www.w3.org/TR/selectors
 
-Here are some examples of XPath expressions and their meanings:
+Here are some examples of XPath expressions, their meanings and CSS
+equivalents:
 
 * ``/html/head/title``: selects the ``<title>`` element, inside the ``<head>``
-  element of an HTML document. Equivalent CSS selector: ``html > head > title``.
+  element of an HTML document. Using CSS, the equivalent would be: ``html >
+  head > title``.
 
 * ``/html/head/title/text()``: selects the text inside the aforementioned
-  ``<title>`` element. Equivalent CSS selector: ``html > head > title ::text``.
+  ``<title>`` element. In Scrapy, you can do the same with CSS using ``html >
+  head > title::text``. The ``::text`` bit isn't really CSS, but is supported
+  by Scrapy for extracting purposes.
 
 * ``//td``: selects all the ``<td>`` elements from the whole document.
-  Equivalent CSS selector: ``td``.
+  The equivalent CSS selector: ``td``.
 
-* ``//div[@class="mine"]``: selects all ``div`` elements which contain an
-  attribute ``class="mine"``. Equivalent CSS selector: ``div.mine``.
+* ``//div[@id="mine"]``: selects all ``div`` elements which contain an
+  attribute ``id="mine"``. The equivalent CSS selector would be: ``div#mine``.
 
-These are just a couple of simple examples of what you can do with XPath, but
-XPath expressions are indeed much more powerful. To learn more about XPath, we
-recommend `this tutorial to learn XPath through examples
-<http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this tutorial to learn "how
-to think in XPath" <http://plasmasturm.org/log/xpath101/>`_.
+These are just a couple of simple examples of what you can do with XPath and
+CSS. XPath expressions are very powerful, they're the foundation of Scrapy
+selectors. In fact, CSS selectors are converted to XPath expressions
+under-the-hood. To learn more about XPath, we recommend `this tutorial to learn
+XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
+tutorial to learn "how to think in XPath"
+<http://plasmasturm.org/log/xpath101/>`_.
 
 .. note:: **CSS vs XPath:** you can go a long way extracting data from web pages
   using only CSS selectors. However, XPath offers more power because besides

From 233b98d642f3c20d47917b54f6944184ee61e0cf Mon Sep 17 00:00:00 2001
From: Valdir Stumm Junior <stummjr@gmail.com>
Date: Fri, 16 Sep 2016 18:08:10 -0300
Subject: [PATCH 09/20] include section describing spider arguments

---
 docs/intro/tutorial.rst | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 34d33a9b3..79cf502a6 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -524,6 +524,46 @@ much because of a programming mistake. This can be configured by the setting
     spider that implements a small rules engine that you can use to write your
     crawlers on top of it.
 
+Customizing behavior via spider arguments
+=========================================
+You can provide command line arguments to your spiders by using the ``-a``
+option when running them::
+
+    scrapy crawl quotes -o items.json -a tag=humor
+
+In this example, the value provided for the ``tag`` argument will be available
+via a spider attribute. Using this, you could make your spider get only quotes
+tagged with a specific tag, building the URL based on the argument::
+
+    import scrapy
+
+
+    class QuotesSpider(scrapy.Spider):
+        name = "quotes"
+
+        def start_requests(self):
+            url = 'http://quotes.toscrape.com/'
+            tag = getattr(self, 'tag', None)
+            if tag is not None:
+                url = url + 'tag/' + tag
+            yield scrapy.Request(url)
+
+        def parse(self, response):
+            for quote in response.css('div.quote'):
+                yield {
+                    'text': quote.css('span.text::text').extract_first(),
+                    'author': quote.css('span small a::text').extract_first(),
+                }
+
+            next_page = response.css('li.next a::attr("href")').extract_first()
+            if next_page is not None:
+                next_page = response.urljoin(next_page)
+                yield scrapy.Request(next_page, callback=self.parse)
+
+
+If you pass the ``tag=humor`` argument to this spider, you'll notice that it
+will only visit URLs from the ``humor`` tag, such as
+``http://quotes.toscrape.com/tag/humor``.
 
 Storing the scraped data
 ========================

From 2a409d1d951d3c968231785cbc1c2c398445888a Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Mon, 19 Sep 2016 17:13:04 -0300
Subject: [PATCH 10/20] [wip] changing introduction to scraping with selectors

---
 docs/intro/tutorial.rst | 291 ++++++++++++----------------------------
 1 file changed, 88 insertions(+), 203 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 79cf502a6..f6aa6476c 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -180,235 +180,120 @@ is Scrapy's default callback method.
 Extracting data
 ---------------
 
-Introduction to Selectors
-^^^^^^^^^^^^^^^^^^^^^^^^^
+The best way to learn how to extract data with Scrapy is trying selectors
+using the shell :ref:`Scrapy shell <topics-shell>`. Run::
 
-There are several ways to extract data from web pages. Scrapy uses a mechanism
-based on `XPath`_ or `CSS`_ expressions called :ref:`Scrapy Selectors
-<topics-selectors>`.  For more information about selectors and other extraction
-mechanisms see the :ref:`Selectors documentation <topics-selectors>`.
+    scrapy crawl http://quotes.toscrape.com/page/1/
 
-.. _XPath: https://www.w3.org/TR/xpath
-.. _CSS: https://www.w3.org/TR/selectors
-
-Here are some examples of XPath expressions, their meanings and CSS
-equivalents:
-
-* ``/html/head/title``: selects the ``<title>`` element, inside the ``<head>``
-  element of an HTML document. Using CSS, the equivalent would be: ``html >
-  head > title``.
-
-* ``/html/head/title/text()``: selects the text inside the aforementioned
-  ``<title>`` element. In Scrapy, you can do the same with CSS using ``html >
-  head > title::text``. The ``::text`` bit isn't really CSS, but is supported
-  by Scrapy for extracting purposes.
-
-* ``//td``: selects all the ``<td>`` elements from the whole document.
-  The equivalent CSS selector: ``td``.
-
-* ``//div[@id="mine"]``: selects all ``div`` elements which contain an
-  attribute ``id="mine"``. The equivalent CSS selector would be: ``div#mine``.
-
-These are just a couple of simple examples of what you can do with XPath and
-CSS. XPath expressions are very powerful, they're the foundation of Scrapy
-selectors. In fact, CSS selectors are converted to XPath expressions
-under-the-hood. To learn more about XPath, we recommend `this tutorial to learn
-XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
-tutorial to learn "how to think in XPath"
-<http://plasmasturm.org/log/xpath101/>`_.
-
-.. note:: **CSS vs XPath:** you can go a long way extracting data from web pages
-  using only CSS selectors. However, XPath offers more power because besides
-  navigating the structure, it can also look at the content: you're
-  able to select things like: *the link that contains the text 'Next Page'*.
-  Because of this, we encourage you to learn about XPath even if you
-  already know how to construct CSS selectors.
-
-For working with CSS and XPath expressions, Scrapy provides the
-:class:`~scrapy.selector.Selector` class and convenient shortcuts to avoid
-instantiating selectors yourself every time you need to select something from a
-response.
-
-You can see selectors as objects that represent nodes in the document
-structure. So, the first instantiated selectors are associated with the root
-node, or the entire document.
-
-Selectors have four basic methods (click on the method to see the complete API
-documentation):
-
-* :meth:`~scrapy.selector.Selector.xpath`: returns a list of selectors, each of
-  which represents the nodes selected by the xpath expression given as
-  argument.
-
-* :meth:`~scrapy.selector.Selector.css`: returns a list of selectors, each of
-  which represents the nodes selected by the CSS expression given as argument.
-
-* :meth:`~scrapy.selector.Selector.extract`: returns a unicode string with the
-  selected data.
-
-* :meth:`~scrapy.selector.Selector.re`: returns a list of unicode strings
-  extracted by applying the regular expression given as argument.
-
-
-Trying Selectors in the Shell
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy
-shell <topics-shell>`, which also requires `IPython <http://ipython.org/>`_ (an extended Python console)
-installed on your system.
-
-To start a shell, you must go to the project's top level directory and run::
-
-    scrapy shell "http://quotes.toscrape.com"
-
-.. note::
-
-   Remember to always enclose urls in quotes when running Scrapy shell from
-   command-line, otherwise urls containing arguments (ie. ``&`` character)
-   will not work.
-
-This is what the shell looks like::
+You will see something like::
 
     [ ... Scrapy log here ... ]
-
-    2016-09-01 18:14:39 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com> (referer: None)
+    2016-09-19 12:09:27 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
     [s] Available Scrapy objects:
-    [s]   crawler    <scrapy.crawler.Crawler object at 0x109001c90>
+    [s]   crawler    <scrapy.crawler.Crawler object at 0x7fa91d888c90>
     [s]   item       {}
-    [s]   request    <GET http://quotes.toscrape.com>
-    [s]   response   <200 http://quotes.toscrape.com>
-    [s]   settings   <scrapy.settings.Settings object at 0x109001610>
-    [s]   spider     <DefaultSpider 'default' at 0x1092808d0>
+    [s]   request    <GET http://quotes.toscrape.com/page/1/>
+    [s]   response   <200 http://quotes.toscrape.com/page/1/>
+    [s]   settings   <scrapy.settings.Settings object at 0x7fa91d888c10>
+    [s]   spider     <DefaultSpider 'default' at 0x7fa91c8af990>
     [s] Useful shortcuts:
     [s]   shelp()           Shell help (print this help)
     [s]   fetch(req_or_url) Fetch request (or URL) and update local objects
     [s]   view(response)    View response in a browser
-    
-    >>>
+    >>> 
 
-After the shell loads, you will have the response fetched in a local
-``response`` variable, so if you type ``response.body`` you will see the body
-of the response, or you can type ``response.headers`` to see its headers.
+Using the shell, you can try selecting elements using `CSS`_ with the response
+object::
 
-More importantly ``response`` has a ``selector`` attribute which is an instance of
-:class:`~scrapy.selector.Selector` class, instantiated with this particular ``response``.
-You can run queries on ``response`` by calling ``response.selector.xpath()`` or
-``response.selector.css()``. There are also some convenience shortcuts like ``response.xpath()``
-or ``response.css()`` which map directly to ``response.selector.xpath()`` and
-``response.selector.css()``.
+    >>> response.css('title')
+    [<Selector xpath=u'descendant-or-self::title' data=u'<title>Quotes to Scrape</title>'>]
 
+The result of running ``response.css('title')`` is a list-like object called
+:class:`~scrapy.selector.SelectorList`, which represents a list of
+:class:`~scrapy.selector.Selector` objects that wrap around XML/HTML elements
+and allow you to run further queries to fine-grain the selection or extract the
+data.
 
-So let's try it::
+To extract the text from the title above, you can do::
 
-    In [1]: response.xpath('//title')
-    Out[1]: [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>] 
-    
-    In [2]: response.xpath('//title').extract()
-    Out[2]: [u'<title>Quotes to Scrape</title>']
-    
-    In [3]: response.xpath('//title/text()')
-    Out[3]: [<Selector xpath='//title/text()' data=u'Quotes to Scrape'>]
+    >>> response.css('title::text').extract()
+    [u'Quotes to Scrape']
 
-    In [4]: response.xpath('//title/text()').extract()
-    Out[4]: [u'Quotes to Scrape']
-    
-    In [11]: response.xpath('//title/text()').re('(\w+)')
-    Out[11]: [u'Quotes', u'to', u'Scrape']
+There are two things to note here: one is that we've added ``::text`` to the
+CSS query, to mean that we want to select the text from inside the title element.
 
-Extracting the data
-^^^^^^^^^^^^^^^^^^^
+The other is that the result of calling ``.extract()`` is a list, because we're
+dealing with an instance :class:`~scrapy.selector.SelectorList`.  When you know
+you just want the first result, as in this case, you can do::
 
-Now, let's try to extract some real information from those pages.
+    >>> response.css('title::text').extract_first()
+    u'Quotes to Scrape'
 
-You could type ``response.body`` in the console, and inspect the source code to
-figure out the XPaths you need to use. However, inspecting the raw HTML code
-there could become a very tedious task. To make it easier, you can
-use Firefox Developer Tools or some Firefox extensions like Firebug. For more
+As an alternative, you could've written::
+
+    >>> response.css('title::text')[0].extract()
+    u'Quotes to Scrape'
+
+However, using ``.extract_first()`` avoids an ``IndexError`` and returns
+``None`` when it doesn't find any element matching the selection.
+
+There's a lesson here: for most scraping code, you want it to be resilient to
+errors due to things not being found on a page, so that even if some parts fail
+to be scraped, you can at least get **some** data.
+
+Besides the :meth:`~scrapy.selector.Selector.extract` and
+:meth:`~scrapy.selector.SelectorList.extract_first` methods, you can also use
+the :meth:`~scrapy.selector.Selector.re` method to extract using a regular
+expression::
+
+    >>> response.css('title::text').re('Quotes.*')
+    [u'Quotes to Scrape']
+    >>> response.css('title::text').re('Q\w+')
+    [u'Quotes']
+    >>> response.css('title::text').re('(\w+) to (\w+)')
+    [u'Quotes', u'Scrape']
+
+In order to find the proper CSS selectors to use, you might find useful opening
+the response page from the shell in your web browser using ``view(response)``.
+You can use your browser developer tools or extensions like Firebug. For more
 information see :ref:`topics-firebug` and :ref:`topics-firefox`.
 
-After inspecting the page source, you'll find that every quote in the website
-is inside a separate ``<div class="quote">`` element, such as::
 
-    <div class="quote">
-        <span class="text">“We accept the love we think we deserve.”</span>
-        <span>by <small class="author">Stephen Chbosky</small></span>
-        <div class="tags">
-            Tags:
-            <meta class="keywords"> 
-            <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
-            <a class="tag" href="/tag/love/page/1/">love</a>
-        </div>
-    </div>
+XPath: a brief intro
+^^^^^^^^^^^^^^^^^^^^
+
+Besides CSS, Scrapy selectors also support using `XPath`_ expressions::
+
+    >>> response.xpath('//title')
+    [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]
+    >>> response.xpath('//title/text()').extract_first()
+    u'Quotes to Scrape'
+
+XPath expressions are very powerful, and are the foundation of Scrapy
+Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
+can see that if you read closely the text representation of the selector
+objects in the shell.
+
+While perhaps not as popular as CSS selectors, XPath expressions offer more
+power because besides navigating the structure, it can also look at the
+content. Using XPath, you're able to select things like: **select the link
+that contains the text "Next Page"**. This makes XPath very fitting to the task
+of scraping, and we encourage you to learn XPath even if you already know how to
+construct CSS selectors, it will make scraping much easier.
+
+We won't cover much of XPath here. To learn more about XPath, we recommend `this tutorial to learn
+XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
+tutorial to learn "how to think in XPath"
+<http://plasmasturm.org/log/xpath101/>`_.
 
 
-So we can select each ``<div class="quote">`` element belonging to the site's 
-list with this code::
+Extraction wrap-up
+^^^^^^^^^^^^^^^^^^
 
-    response.xpath('//div[@class="quote"]')
+Now that you know a bit about selection and extraction, let's complete our
+spider by writing the code to extract the quotes from the webpage.
 
-From the quote elements, we can select the texts with::
-
-    response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
-
-The authors::
-
-    response.xpath('//div[@class="quote"]/span/small/text()').extract()
-
-As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
-concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
-that property here, so::
-
-    for quote in response.xpath('//div[@class="quote"]'):
-        text = quote.xpath('span[@class="text"]/text()').extract_first()
-        author = quote.xpath('span/small/text()').extract_first()
-        print({'text': text, 'author': author})
-
-In the above snippet we've decided to use the method ``.extract_first()``
-instead of ``.extract()``, to extract the content from the first element from a
-selector list returned by ``.xpath()``.
-
-.. note::
-
-    For a more detailed description of using nested selectors, see
-    :ref:`topics-selectors-nesting-selectors` and
-    :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors`
-    documentation
-
-Knowing to use selectors, extracting data from a page is just a matter of
-yield the Python dictionaries from the callback method instead of printing
-them.
-
-Let's add the necessary code to our spider::
-
-    import scrapy
-
-
-    class QuotesSpider(scrapy.Spider):
-        name = "quotes"
-        start_urls = [
-            'http://quotes.toscrape.com/page/1/',
-            'http://quotes.toscrape.com/page/2/',
-        ]
-
-        def parse(self, response):
-            for quote in response.xpath('//div[@class="quote"]'):
-                yield {
-                    'text': quote.xpath('span[@class="text"]/text()').extract_first(),
-                    'author': quote.xpath('span/small/text()').extract_first(),
-                }
-
-Run::
-
-    scrapy crawl quotes
-
-Now crawling quotes.toscrape.com will show dictionary objects::
-
-    2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
-    {'author': 'Oscar Wilde',
-     'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
-    2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
-    {'author': 'Mark Twain',
-     'text': '“The man who does not read has no advantage over the man who cannot read.”'}
+TODO: show how to extract quotes and integrate spider code here.
 
 
 Following links

From fee07835f2f9504acc7f0952088ca2ea201027c3 Mon Sep 17 00:00:00 2001
From: Valdir Stumm Junior <stummjr@gmail.com>
Date: Mon, 19 Sep 2016 19:19:31 -0300
Subject: [PATCH 11/20] Completing the data extraction section

---
 docs/intro/tutorial.rst | 122 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 112 insertions(+), 10 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f6aa6476c..473183be3 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -106,8 +106,8 @@ and defines some attributes and methods:
   the scraped data as dicts and also finding new URLs to
   follow and creating new requests (:class:`~scrapy.http.Request`) from them.
 
-How to run your spider
-----------------------
+How to run our spider
+---------------------
 
 To put our spider to work, go to the project's top level directory and run::
 
@@ -148,12 +148,14 @@ objects and calls the ``parse`` callback method passing the response as
 argument.
 
 
-Simplifying your spider
------------------------
-Instead of defining the :meth:`~scrapy.spiders.Spider.start_requests` method
-generating :class:`scrapy.Request <scrapy.http.Request>`
-objects from URLs, you can just put those URLs in the 
-:attr:`~scrapy.spiders.Spider.start_urls` attribute::
+A shortcut to the start_requests method
+---------------------------------------
+Instead of implementing a :meth:`~scrapy.spiders.Spider.start_requests` method
+that generates :class:`scrapy.Request <scrapy.http.Request>` objects from URLs,
+you can just define a :attr:`~scrapy.spiders.Spider.start_urls` class attribute
+with a list of URLs. This list will then be used by the default implementation
+of :meth:`~scrapy.spiders.Spider.start_requests` to create the initial requests
+for your spider::
 
     import scrapy
 
@@ -174,7 +176,8 @@ objects from URLs, you can just put those URLs in the
 The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle
 each of the requests for those URLs, even though we haven't explicitely told
 Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse`
-is Scrapy's default callback method.
+is Scrapy's default callback method that is called for any request that have
+been generated with no callback explicitely assigned to handle it.
 
 
 Extracting data
@@ -293,7 +296,104 @@ Extraction wrap-up
 Now that you know a bit about selection and extraction, let's complete our
 spider by writing the code to extract the quotes from the webpage.
 
-TODO: show how to extract quotes and integrate spider code here.
+Each quote in http://quotes.toscrape.com is represented by HTML code that looks
+like this::
+
+    <div class="quote">
+        <span class="text">“The world as we have created it is a process of our
+        thinking. It cannot be changed without changing our thinking.”</span>
+        <span>
+            by <small class="author">Albert Einstein</small>
+            <a href="/author/Albert-Einstein">(about)</a>
+        </span>
+        <div class="tags">
+            Tags:
+            <a class="tag" href="/tag/change/page/1/">change</a>
+            <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
+            <a class="tag" href="/tag/thinking/page/1/">thinking</a>
+            <a class="tag" href="/tag/world/page/1/">world</a>
+        </div>
+    </div>
+
+Let's open up scrapy shell and play a bit to find out how to extract the data
+we want::
+
+    $ scrapy shell http://quotes.toscrape.com
+
+We get a list of selectors to the quotes using::
+
+    >>> response.css("div.quote")
+
+Each of the selectors returned by the query above allows us to run further
+queries over the quotes itselves. Let's assign the first selector to a
+variable, so that we can run our CSS selectors directly on a particular quote::
+
+    >>> quote = response.css("div.quote")[0]
+
+Now, let's extract ``title``, ``author`` and the ``tags`` from that quote
+using the ``quote`` object we just created::
+
+    >>> title = quote.css("span.text ::text").extract_first()
+    >>> title
+    '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
+    >>> author = quote.css("small.author ::text").extract_first()
+    >>> author
+    'Albert Einstein'
+
+Given that the tags is a list of strings, we can use the ``.extract()`` method
+to get all of them::
+
+    >>> tags = quote.css("div.tags a.tag ::text").extract()
+    >>> tags
+    ['change', 'deep-thoughts', 'thinking', 'world']
+
+Now, we can iterate over all the quotes in the page and use the CSS selectors
+we defined to extract data::
+
+    >>> for quote in response.css("div.quote"):
+    ...     text = quote.css("span.text ::text").extract_first()
+    ...     author = quote.css("small.author ::text").extract_first()
+    ...     tags = quote.css("div.tags a.tag ::text").extract()
+    ...     print("{} - {} - {}".format(text, author, tags))
+
+
+Extracting data in our spider
+------------------------------
+
+Until now, the spider we built doesn't extract any data in particular. I just
+saves the whole HTML page to a local file. Now, let's integrate the extraction
+logic above in our spider.
+
+A Scrapy spider typically generates many dictionaries containing the data
+extracted from the page. To do that, we use the ``yield`` Python keyword, as
+you can see below::
+
+    import scrapy
+
+
+    class QuotesSpider(scrapy.Spider):
+        name = "quotes"
+        start_urls = [
+            'http://quotes.toscrape.com/page/1/',
+            'http://quotes.toscrape.com/page/2/',
+        ]
+
+        def parse(self, response):
+            for quote in response.css('div.quote'):
+                yield {
+                    'text': quote.css('span.text::text').extract_first(),
+                    'author': quote.css('span small::text').extract_first(),
+                    'tags': quote.css("div.tags a.tag ::text").extract(),
+                }
+
+If you run this spider, it will output the extracted data with the log::
+
+    2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
+    {'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
+    2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
+    {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}
+
+:ref:`Later in the tutorial <storing-data>`, we will see how to save this data to a file.
 
 
 Following links
@@ -450,6 +550,8 @@ If you pass the ``tag=humor`` argument to this spider, you'll notice that it
 will only visit URLs from the ``humor`` tag, such as
 ``http://quotes.toscrape.com/tag/humor``.
 
+.. _storing-data:
+
 Storing the scraped data
 ========================
 

From f4f93c5c266648317ff2c2e474b7d5fd08918c07 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Tue, 20 Sep 2016 09:19:48 -0300
Subject: [PATCH 12/20] fix tox docs build, adjust title

---
 docs/intro/tutorial.rst | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 473183be3..6a5e99d7a 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -265,7 +265,7 @@ information see :ref:`topics-firebug` and :ref:`topics-firefox`.
 XPath: a brief intro
 ^^^^^^^^^^^^^^^^^^^^
 
-Besides CSS, Scrapy selectors also support using `XPath`_ expressions::
+Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions::
 
     >>> response.xpath('//title')
     [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]
@@ -289,6 +289,8 @@ XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
 tutorial to learn "how to think in XPath"
 <http://plasmasturm.org/log/xpath101/>`_.
 
+.. _XPath: https://www.w3.org/TR/xpath
+.. _CSS: https://www.w3.org/TR/selectors
 
 Extraction wrap-up
 ^^^^^^^^^^^^^^^^^^
@@ -453,7 +455,7 @@ using a :ref:`trick to pass additional data to the callbacks
 <topics-request-response-ref-request-callback-arguments>`.
 
 Another example: scraping authors
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+---------------------------------
 
 Here is another spider that illustrates callbacks and following links,
 this time for scraping author information::
@@ -509,8 +511,9 @@ much because of a programming mistake. This can be configured by the setting
     spider that implements a small rules engine that you can use to write your
     crawlers on top of it.
 
-Customizing behavior via spider arguments
-=========================================
+Adding a spider argument
+========================
+
 You can provide command line arguments to your spiders by using the ``-a``
 option when running them::
 

From 125b691102320864c635608618636253074eae1a Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Tue, 20 Sep 2016 12:47:03 -0300
Subject: [PATCH 13/20] more reviewing and editing, minor restructure, syntax
 fixes

---
 docs/intro/tutorial.rst | 141 ++++++++++++++++++++++++----------------
 1 file changed, 84 insertions(+), 57 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 6a5e99d7a..b4a5e3b48 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -14,7 +14,9 @@ This tutorial will walk you through these tasks:
 
 1. Creating a new Scrapy project
 2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract data
-3. Exporting the scraped data using command line
+3. Exporting the scraped data using the command line
+4. Change spider to recursively follow links
+5. Using spider arguments
 
 Scrapy is written in Python_. If you're new to the language you might want to
 start by getting an idea of what the language is like, to get the most out of
@@ -43,7 +45,7 @@ This will create a ``tutorial`` directory with the following contents::
         tutorial/             # project's Python module, you'll import your code from here
             __init__.py
 
-            items.py          # project items file
+            items.py          # project items definition file
 
             pipelines.py      # project pipelines file
 
@@ -109,7 +111,8 @@ and defines some attributes and methods:
 How to run our spider
 ---------------------
 
-To put our spider to work, go to the project's top level directory and run::
+To put our spider to work, go to the project's top level directory (``cd
+tutorial``) and run::
 
    scrapy crawl quotes
 
@@ -141,6 +144,7 @@ for the respective URLs, as our ``parse`` method instructs.
 
 What just happened under the hood?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
 returned by the ``start_requests`` method of the Spider. Upon receiving
 a response for each one, it instantiates :class:`scrapy.http.Response`
@@ -173,11 +177,11 @@ for your spider::
             with open(filename, 'wb') as f:
                 f.write(response.body)
 
-The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle
-each of the requests for those URLs, even though we haven't explicitely told
-Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse`
-is Scrapy's default callback method that is called for any request that have
-been generated with no callback explicitely assigned to handle it.
+The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each
+of the requests for those URLs, even though we haven't explicitely told Scrapy
+to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's
+default callback method, which is called for requests without an explicitely
+assigned callback.
 
 
 Extracting data
@@ -224,10 +228,14 @@ To extract the text from the title above, you can do::
 
 There are two things to note here: one is that we've added ``::text`` to the
 CSS query, to mean that we want to select the text from inside the title element.
+If we don't specify ``::text``, we'd get the HTML tags::
 
-The other is that the result of calling ``.extract()`` is a list, because we're
-dealing with an instance :class:`~scrapy.selector.SelectorList`.  When you know
-you just want the first result, as in this case, you can do::
+    >>> response.css('title').extract()
+    [u'<title>Quotes to Scrape</title>']
+
+The other thing is that the result of calling ``.extract()`` is a list, because
+we're dealing with an instance of :class:`~scrapy.selector.SelectorList`.  When
+you know you just want the first result, as in this case, you can do::
 
     >>> response.css('title::text').extract_first()
     u'Quotes to Scrape'
@@ -284,22 +292,24 @@ that contains the text "Next Page"**. This makes XPath very fitting to the task
 of scraping, and we encourage you to learn XPath even if you already know how to
 construct CSS selectors, it will make scraping much easier.
 
-We won't cover much of XPath here. To learn more about XPath, we recommend `this tutorial to learn
-XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
-tutorial to learn "how to think in XPath"
-<http://plasmasturm.org/log/xpath101/>`_.
+We won't cover much of XPath here. To learn more about XPath, we recommend
+`this tutorial to learn XPath through examples
+<http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this tutorial to learn "how
+to think in XPath" <http://plasmasturm.org/log/xpath101/>`_.
 
 .. _XPath: https://www.w3.org/TR/xpath
 .. _CSS: https://www.w3.org/TR/selectors
 
-Extraction wrap-up
-^^^^^^^^^^^^^^^^^^
+Extracting quotes and authors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Now that you know a bit about selection and extraction, let's complete our
 spider by writing the code to extract the quotes from the webpage.
 
-Each quote in http://quotes.toscrape.com is represented by HTML code that looks
-like this::
+Each quote in http://quotes.toscrape.com is represented by HTML elements that look
+like this:
+
+.. code-block:: html
 
     <div class="quote">
         <span class="text">“The world as we have created it is a process of our
@@ -322,12 +332,12 @@ we want::
 
     $ scrapy shell http://quotes.toscrape.com
 
-We get a list of selectors to the quotes using::
+We get a list of selectors for the quote HTML elements with::
 
     >>> response.css("div.quote")
 
 Each of the selectors returned by the query above allows us to run further
-queries over the quotes itselves. Let's assign the first selector to a
+queries over their sub-elements. Let's assign the first selector to a
 variable, so that we can run our CSS selectors directly on a particular quote::
 
     >>> quote = response.css("div.quote")[0]
@@ -342,33 +352,33 @@ using the ``quote`` object we just created::
     >>> author
     'Albert Einstein'
 
-Given that the tags is a list of strings, we can use the ``.extract()`` method
+Given that the tags are a list of strings, we can use the ``.extract()`` method
 to get all of them::
 
     >>> tags = quote.css("div.tags a.tag ::text").extract()
     >>> tags
     ['change', 'deep-thoughts', 'thinking', 'world']
 
-Now, we can iterate over all the quotes in the page and use the CSS selectors
-we defined to extract data::
+Having figured out how to extract each bit, we can now iterate over all the
+quotes elements and put them together into a Python dictionary::
 
     >>> for quote in response.css("div.quote"):
     ...     text = quote.css("span.text ::text").extract_first()
     ...     author = quote.css("small.author ::text").extract_first()
     ...     tags = quote.css("div.tags a.tag ::text").extract()
-    ...     print("{} - {} - {}".format(text, author, tags))
+    ...     print(dict(text=text, author=author, tags=tags))
 
 
 Extracting data in our spider
 ------------------------------
 
-Until now, the spider we built doesn't extract any data in particular. I just
-saves the whole HTML page to a local file. Now, let's integrate the extraction
-logic above in our spider.
+Let's get back to our spider. Until now, it doesn't extract any data in
+particular, just saves the whole HTML page to a local file. Let's integrate the
+extraction logic above into our spider.
 
 A Scrapy spider typically generates many dictionaries containing the data
-extracted from the page. To do that, we use the ``yield`` Python keyword, as
-you can see below::
+extracted from the page. To do that, we use the ``yield`` Python keyword
+in the callback, as you can see below::
 
     import scrapy
 
@@ -395,7 +405,38 @@ If you run this spider, it will output the extracted data with the log::
     2016-09-19 18:57:19 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
     {'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}
 
-:ref:`Later in the tutorial <storing-data>`, we will see how to save this data to a file.
+
+.. _storing-data:
+
+Storing the scraped data
+========================
+
+The simplest way to store the scraped data is by using :ref:`Feed exports
+<topics-feed-exports>`, with the following command::
+
+    scrapy crawl quotes -o items.json
+
+That will generate an ``items.json`` file containing all scraped items,
+serialized in `JSON`_.
+
+You could've also used other formats, like `JSON Lines`_::
+
+    scrapy crawl quotes -o items.jl
+
+The `JSON Lines`_ format is useful because it's stream-like, you can easily
+append new records to it. As each record is a separate line, you can also
+process big files without having to fit everything in memory, there are tools
+like `JQ`_ to help doing that at the command-line.
+
+In small projects (like the one in this tutorial), that should be enough.
+However, if you want to perform more complex things with the scraped items, you
+can write an :ref:`Item Pipeline <topics-item-pipeline>`. As with Items, a
+placeholder file for Item Pipelines has been set up for you when the project is
+created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
+pipelines if you just want to store the scraped items.
+
+.. _JSON Lines: http://jsonlines.org
+.. _JQ: https://stedolan.github.io/jq
 
 
 Following links
@@ -511,17 +552,20 @@ much because of a programming mistake. This can be configured by the setting
     spider that implements a small rules engine that you can use to write your
     crawlers on top of it.
 
-Adding a spider argument
-========================
+Using spider arguments
+======================
 
 You can provide command line arguments to your spiders by using the ``-a``
 option when running them::
 
     scrapy crawl quotes -o items.json -a tag=humor
 
+These arguments are passed to the Spider's ``__init__`` method and become
+spider attributes by default.  
+
 In this example, the value provided for the ``tag`` argument will be available
-via a spider attribute. Using this, you could make your spider get only quotes
-tagged with a specific tag, building the URL based on the argument::
+via ``self.tag``. You can use this to make your spider fetch only quotes
+with a specific tag, building the URL based on the argument::
 
     import scrapy
 
@@ -553,25 +597,7 @@ If you pass the ``tag=humor`` argument to this spider, you'll notice that it
 will only visit URLs from the ``humor`` tag, such as
 ``http://quotes.toscrape.com/tag/humor``.
 
-.. _storing-data:
-
-Storing the scraped data
-========================
-
-The simplest way to store the scraped data is by using :ref:`Feed exports
-<topics-feed-exports>`, with the following command::
-
-    scrapy crawl quotes -o items.json
-
-That will generate an ``items.json`` file containing all scraped items,
-serialized in `JSON`_.
-
-In small projects (like the one in this tutorial), that should be enough.
-However, if you want to perform more complex things with the scraped items, you
-can write an :ref:`Item Pipeline <topics-item-pipeline>`. As with Items, a
-placeholder file for Item Pipelines has been set up for you when the project is
-created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
-pipelines if you just want to store the scraped items.
+You can :ref:`learn more about handling spider arguments here <spiderargs>`.
 
 Next steps
 ==========
@@ -580,9 +606,10 @@ This tutorial covered only the basics of Scrapy, but there's a lot of other
 features not mentioned here. Check the :ref:`topics-whatelse` section in
 :ref:`intro-overview` chapter for a quick overview of the most important ones.
 
-Then, we recommend you continue by playing with an example project (see
-:ref:`intro-examples`), and then continue with the section
-:ref:`section-basics`.
+You can continue from the section :ref:`section-basics` to know more about the
+command-line tool, spiders and other things the tutorial haven't covered like
+modeling the scraped data. If you prefer to play with an example project, check
+the :ref:`intro-examples` section.
 
 .. _JSON: https://en.wikipedia.org/wiki/JSON
 .. _dirbot: https://github.com/scrapy/dirbot

From bc41fdf20e76ec06677a3d488323796ff2e126f7 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Tue, 20 Sep 2016 15:04:08 -0300
Subject: [PATCH 14/20] address review comments, add debug log to initial
 spider

---
 docs/intro/tutorial.rst | 94 ++++++++++++++++++++++++++---------------
 1 file changed, 61 insertions(+), 33 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index b4a5e3b48..162fa242e 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -15,7 +15,7 @@ This tutorial will walk you through these tasks:
 1. Creating a new Scrapy project
 2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract data
 3. Exporting the scraped data using the command line
-4. Change spider to recursively follow links
+4. Changing spider to recursively follow links
 5. Using spider arguments
 
 Scrapy is written in Python_. If you're new to the language you might want to
@@ -60,9 +60,9 @@ Our first Spider
 
 Spiders are classes that you define and that Scrapy uses to scrape information
 from a website (or group of websites). They must subclass
-:class:`scrapy.Spider` and define the initial requests to make, how to follow
-links in the pages, and how to parse the downloaded page content to extract
-data.
+:class:`scrapy.Spider` and define the initial requests to make, optionally how
+to follow links in the pages, and how to parse the downloaded page content to
+extract data.
 
 This is the code for our first Spider. Save it in a file named
 ``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::
@@ -86,6 +86,7 @@ This is the code for our first Spider. Save it in a file named
             filename = 'quotes-%s.html' % page
             with open(filename, 'wb') as f:
                 f.write(response.body)
+            self.log('Saved file %s' % filename)
 
 
 As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>`
@@ -120,19 +121,17 @@ This command runs the spider with name ``quotes`` that we've just added, that
 will send some requests for the ``quotes.toscrape.com`` domain. You will get an output
 similar to this::
 
-
-    2016-09-01 16:51:27 [scrapy] INFO: Scrapy started (bot: tutorial)
-    2016-09-01 16:51:27 [scrapy] INFO: Overridden settings: {...}
-    2016-09-01 16:51:27 [scrapy] INFO: Enabled extensions: ...
-    2016-09-01 16:51:27 [scrapy] INFO: Enabled downloader middlewares: ...
-    2016-09-01 16:51:27 [scrapy] INFO: Enabled spider middlewares: ...
-    2016-09-01 16:51:27 [scrapy] INFO: Enabled item pipelines: ...
-    2016-09-01 16:51:27 [scrapy] INFO: Spider opened
-    2016-09-01 16:51:27 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
-    2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
-    2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
-    2016-09-01 16:51:29 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
-    2016-09-01 16:51:29 [scrapy] INFO: Closing spider (finished)
+    ... (omitted for brevity)
+    2016-09-20 14:48:00 [scrapy] INFO: Spider opened
+    2016-09-20 14:48:00 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
+    2016-09-20 14:48:00 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
+    2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
+    2016-09-20 14:48:00 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
+    2016-09-20 14:48:01 [quotes] DEBUG: Saved file quotes-1.html
+    2016-09-20 14:48:01 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
+    2016-09-20 14:48:01 [quotes] DEBUG: Saved file quotes-2.html
+    2016-09-20 14:48:01 [scrapy] INFO: Closing spider (finished)
+    ...
 
 Now, check the files in the current directory. You should notice that two new
 files have been created: *quotes-1.html* and *quotes-2.html*, with the content
@@ -178,9 +177,9 @@ for your spider::
                 f.write(response.body)
 
 The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each
-of the requests for those URLs, even though we haven't explicitely told Scrapy
+of the requests for those URLs, even though we haven't explicitly told Scrapy
 to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's
-default callback method, which is called for requests without an explicitely
+default callback method, which is called for requests without an explicitly
 assigned callback.
 
 
@@ -190,7 +189,7 @@ Extracting data
 The best way to learn how to extract data with Scrapy is trying selectors
 using the shell :ref:`Scrapy shell <topics-shell>`. Run::
 
-    scrapy crawl http://quotes.toscrape.com/page/1/
+    scrapy shell http://quotes.toscrape.com/page/1/
 
 You will see something like::
 
@@ -304,7 +303,7 @@ Extracting quotes and authors
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Now that you know a bit about selection and extraction, let's complete our
-spider by writing the code to extract the quotes from the webpage.
+spider by writing the code to extract the quotes from the web page.
 
 Each quote in http://quotes.toscrape.com is represented by HTML elements that look
 like this:
@@ -345,17 +344,17 @@ variable, so that we can run our CSS selectors directly on a particular quote::
 Now, let's extract ``title``, ``author`` and the ``tags`` from that quote
 using the ``quote`` object we just created::
 
-    >>> title = quote.css("span.text ::text").extract_first()
+    >>> title = quote.css("span.text::text").extract_first()
     >>> title
     '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
-    >>> author = quote.css("small.author ::text").extract_first()
+    >>> author = quote.css("small.author::text").extract_first()
     >>> author
     'Albert Einstein'
 
 Given that the tags are a list of strings, we can use the ``.extract()`` method
 to get all of them::
 
-    >>> tags = quote.css("div.tags a.tag ::text").extract()
+    >>> tags = quote.css("div.tags a.tag::text").extract()
     >>> tags
     ['change', 'deep-thoughts', 'thinking', 'world']
 
@@ -363,10 +362,14 @@ Having figured out how to extract each bit, we can now iterate over all the
 quotes elements and put them together into a Python dictionary::
 
     >>> for quote in response.css("div.quote"):
-    ...     text = quote.css("span.text ::text").extract_first()
-    ...     author = quote.css("small.author ::text").extract_first()
-    ...     tags = quote.css("div.tags a.tag ::text").extract()
+    ...     text = quote.css("span.text::text").extract_first()
+    ...     author = quote.css("small.author::text").extract_first()
+    ...     tags = quote.css("div.tags a.tag::text").extract()
     ...     print(dict(text=text, author=author, tags=tags))
+    {'text': u'\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d', 'tags': [u'change', u'deep-thoughts', u'thinking', u'world'], 'author': u'Albert Einstein'}
+    {'text': u'\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d', 'tags': [u'abilities', u'choices'], 'author': u'J.K. Rowling'}
+        ... a few more of these, omitted for brevity
+    >>>
 
 
 Extracting data in our spider
@@ -395,7 +398,7 @@ in the callback, as you can see below::
                 yield {
                     'text': quote.css('span.text::text').extract_first(),
                     'author': quote.css('span small::text').extract_first(),
-                    'tags': quote.css("div.tags a.tag ::text").extract(),
+                    'tags': quote.css("div.tags a.tag::text").extract(),
                 }
 
 If you run this spider, it will output the extracted data with the log::
@@ -430,9 +433,9 @@ like `JQ`_ to help doing that at the command-line.
 
 In small projects (like the one in this tutorial), that should be enough.
 However, if you want to perform more complex things with the scraped items, you
-can write an :ref:`Item Pipeline <topics-item-pipeline>`. As with Items, a
-placeholder file for Item Pipelines has been set up for you when the project is
-created, in ``tutorial/pipelines.py``. Though you don't need to implement any item
+can write an :ref:`Item Pipeline <topics-item-pipeline>`. A placeholder file
+for Item Pipelines has been set up for you when the project is created, in
+``tutorial/pipelines.py``. Though you don't need to implement any item
 pipelines if you just want to store the scraped items.
 
 .. _JSON Lines: http://jsonlines.org
@@ -448,7 +451,31 @@ from http://quotes.toscrape.com, you want quotes from all the pages in the websi
 Now that you know how to extract data from pages, let's see how to follow links
 from them.
 
-Here is a modification of our spider that recursively follows the link to the next
+First thing is to extract the link to the page we want to follow.  Examining
+our page, we can see there is a link to the next page with the following
+markup:
+
+.. code-block:: html
+
+    <ul class="pager">
+        <li class="next">
+            <a href="/page/2/">Next <span aria-hidden="true">&rarr;</span></a>
+        </li>
+    </ul>
+
+We can try extracting it in the shell::
+
+    >>> response.css('li.next a').extract_first()
+    u'<a href="/page/2/">Next <span aria-hidden="true">\u2192</span></a>'
+
+This gets the anchor element, but we want the attribute ``href``. For that,
+Scrapy supports a CSS extension that let's you select the attribute contents,
+like this::
+
+    >>> response.css('li.next a::attr("href")').extract_first()
+    u'/page/2/'
+
+Let's see now our spider modified to recursively follows the link to the next
 page, extracting data from it::
 
     import scrapy
@@ -465,6 +492,7 @@ page, extracting data from it::
                 yield {
                     'text': quote.css('span.text::text').extract_first(),
                     'author': quote.css('span small::text').extract_first(),
+                    'tags': quote.css("div.tags a.tag::text").extract(),
                 }
 
             next_page = response.css('li.next a::attr("href")').extract_first()
@@ -534,7 +562,7 @@ this time for scraping author information::
 
 This spider will start from the main page, it will follow all the links to the
 authors pages calling the ``parse_author`` callback for each of them, and also
-the pagination links too with the ``parse`` callback as we saw before.
+the pagination links with the ``parse`` callback as we saw before.
 
 The ``parse_author`` callback defines a helper function to extract and cleanup the
 data from a CSS query and yields the Python dict with the author data.

From a876ea5bd2911d5f7d06dfbb4ddbbdde8c51bc27 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Tue, 20 Sep 2016 15:10:49 -0300
Subject: [PATCH 15/20] minor grammar fix

---
 docs/intro/tutorial.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 162fa242e..f4b2d0693 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -475,7 +475,7 @@ like this::
     >>> response.css('li.next a::attr("href")').extract_first()
     u'/page/2/'
 
-Let's see now our spider modified to recursively follows the link to the next
+Let's see now our spider modified to recursively follow the link to the next
 page, extracting data from it::
 
     import scrapy

From c126c593619ebbab6367f22557a44df93486942f Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Tue, 20 Sep 2016 18:19:25 -0300
Subject: [PATCH 16/20] address more review comments

---
 docs/intro/tutorial.rst | 119 +++++++++++++++++++++++-----------------
 1 file changed, 69 insertions(+), 50 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f4b2d0693..65746c389 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -59,7 +59,7 @@ Our first Spider
 ================
 
 Spiders are classes that you define and that Scrapy uses to scrape information
-from a website (or group of websites). They must subclass
+from a website (or a group of websites). They must subclass
 :class:`scrapy.Spider` and define the initial requests to make, optionally how
 to follow links in the pages, and how to parse the downloaded page content to
 extract data.
@@ -96,7 +96,7 @@ and defines some attributes and methods:
   unique within a project, that is, you can't set the same name for different
   Spiders.
 
-* :meth:`~scrapy.spiders.Spider.start_requests`: must return a list
+* :meth:`~scrapy.spiders.Spider.start_requests`: must generate or return a list
   of requests where the Spider will begin to crawl from.
   Subsequent requests will be generated successively from these initial requests.
 
@@ -112,8 +112,7 @@ and defines some attributes and methods:
 How to run our spider
 ---------------------
 
-To put our spider to work, go to the project's top level directory (``cd
-tutorial``) and run::
+To put our spider to work, go to the project's top level directory and run::
 
    scrapy crawl quotes
 
@@ -145,10 +144,10 @@ What just happened under the hood?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
-returned by the ``start_requests`` method of the Spider. Upon receiving
-a response for each one, it instantiates :class:`scrapy.http.Response`
-objects and calls the ``parse`` callback method passing the response as
-argument.
+returned by the ``start_requests`` method of the Spider. Upon receiving a
+response for each one, it instantiates :class:`scrapy.http.Response` objects
+and calls the callback method associated with the request (in this case, the
+``parse`` method) passing the response as argument.
 
 
 A shortcut to the start_requests method
@@ -166,8 +165,8 @@ for your spider::
     class QuotesSpider(scrapy.Spider):
         name = "quotes"
         start_urls = [
-          'http://quotes.toscrape.com/page/1/',
-          'http://quotes.toscrape.com/page/2/',
+            'http://quotes.toscrape.com/page/1/',
+            'http://quotes.toscrape.com/page/2/',
         ]
 
         def parse(self, response):
@@ -189,13 +188,20 @@ Extracting data
 The best way to learn how to extract data with Scrapy is trying selectors
 using the shell :ref:`Scrapy shell <topics-shell>`. Run::
 
-    scrapy shell http://quotes.toscrape.com/page/1/
+    scrapy shell 'http://quotes.toscrape.com/page/1/'
+
+.. note::
+
+   Remember to always enclose urls in quotes when running Scrapy shell from
+   command-line, otherwise urls containing arguments (ie. ``&`` character)
+   will not work.
 
 You will see something like::
 
     [ ... Scrapy log here ... ]
     2016-09-19 12:09:27 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
     [s] Available Scrapy objects:
+    [s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
     [s]   crawler    <scrapy.crawler.Crawler object at 0x7fa91d888c90>
     [s]   item       {}
     [s]   request    <GET http://quotes.toscrape.com/page/1/>
@@ -212,7 +218,7 @@ Using the shell, you can try selecting elements using `CSS`_ with the response
 object::
 
     >>> response.css('title')
-    [<Selector xpath=u'descendant-or-self::title' data=u'<title>Quotes to Scrape</title>'>]
+    [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
 
 The result of running ``response.css('title')`` is a list-like object called
 :class:`~scrapy.selector.SelectorList`, which represents a list of
@@ -223,26 +229,27 @@ data.
 To extract the text from the title above, you can do::
 
     >>> response.css('title::text').extract()
-    [u'Quotes to Scrape']
+    ['Quotes to Scrape']
 
 There are two things to note here: one is that we've added ``::text`` to the
-CSS query, to mean that we want to select the text from inside the title element.
-If we don't specify ``::text``, we'd get the HTML tags::
+CSS query, to mean we want to select only the text elements directly inside
+``<title>`` element.  If we don't specify ``::text``, we'd get the full title
+element, including its tags::
 
     >>> response.css('title').extract()
-    [u'<title>Quotes to Scrape</title>']
+    ['<title>Quotes to Scrape</title>']
 
 The other thing is that the result of calling ``.extract()`` is a list, because
 we're dealing with an instance of :class:`~scrapy.selector.SelectorList`.  When
 you know you just want the first result, as in this case, you can do::
 
     >>> response.css('title::text').extract_first()
-    u'Quotes to Scrape'
+    'Quotes to Scrape'
 
 As an alternative, you could've written::
 
     >>> response.css('title::text')[0].extract()
-    u'Quotes to Scrape'
+    'Quotes to Scrape'
 
 However, using ``.extract_first()`` avoids an ``IndexError`` and returns
 ``None`` when it doesn't find any element matching the selection.
@@ -253,21 +260,27 @@ to be scraped, you can at least get **some** data.
 
 Besides the :meth:`~scrapy.selector.Selector.extract` and
 :meth:`~scrapy.selector.SelectorList.extract_first` methods, you can also use
-the :meth:`~scrapy.selector.Selector.re` method to extract using a regular
-expression::
+the :meth:`~scrapy.selector.Selector.re` method to extract using `regular
+expressions`::
 
-    >>> response.css('title::text').re('Quotes.*')
-    [u'Quotes to Scrape']
-    >>> response.css('title::text').re('Q\w+')
-    [u'Quotes']
-    >>> response.css('title::text').re('(\w+) to (\w+)')
-    [u'Quotes', u'Scrape']
+    >>> response.css('title::text').re(r'Quotes.*')
+    ['Quotes to Scrape']
+    >>> response.css('title::text').re(r'Q\w+')
+    ['Quotes']
+    >>> response.css('title::text').re(r'(\w+) to (\w+)')
+    ['Quotes', 'Scrape']
 
 In order to find the proper CSS selectors to use, you might find useful opening
 the response page from the shell in your web browser using ``view(response)``.
-You can use your browser developer tools or extensions like Firebug. For more
+You can use your browser developer tools or extensions like Firebug.  For more
 information see :ref:`topics-firebug` and :ref:`topics-firefox`.
 
+`Selector Gadget`_ is also a nice tool to quickly find CSS selector for
+visually selected elements.
+
+.. _regular expressions: https://docs.python.org/3/library/re.html
+.. _Selector Gadget: http://selectorgadget.com/
+
 
 XPath: a brief intro
 ^^^^^^^^^^^^^^^^^^^^
@@ -275,9 +288,9 @@ XPath: a brief intro
 Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions::
 
     >>> response.xpath('//title')
-    [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]
+    [<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
     >>> response.xpath('//title/text()').extract_first()
-    u'Quotes to Scrape'
+    'Quotes to Scrape'
 
 XPath expressions are very powerful, and are the foundation of Scrapy
 Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
@@ -291,8 +304,9 @@ that contains the text "Next Page"**. This makes XPath very fitting to the task
 of scraping, and we encourage you to learn XPath even if you already know how to
 construct CSS selectors, it will make scraping much easier.
 
-We won't cover much of XPath here. To learn more about XPath, we recommend
-`this tutorial to learn XPath through examples
+We won't cover much of XPath here, but you can read more about `using XPath
+with Scrapy Selectors here <topics-selectors>`_. To learn more about XPath, we
+recommend `this tutorial to learn XPath through examples
 <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this tutorial to learn "how
 to think in XPath" <http://plasmasturm.org/log/xpath101/>`_.
 
@@ -366,8 +380,8 @@ quotes elements and put them together into a Python dictionary::
     ...     author = quote.css("small.author::text").extract_first()
     ...     tags = quote.css("div.tags a.tag::text").extract()
     ...     print(dict(text=text, author=author, tags=tags))
-    {'text': u'\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d', 'tags': [u'change', u'deep-thoughts', u'thinking', u'world'], 'author': u'Albert Einstein'}
-    {'text': u'\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d', 'tags': [u'abilities', u'choices'], 'author': u'J.K. Rowling'}
+    {'tags': ['change', 'deep-thoughts', 'thinking', 'world'], 'author': 'Albert Einstein', 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}
+    {'tags': ['abilities', 'choices'], 'author': 'J.K. Rowling', 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”'}
         ... a few more of these, omitted for brevity
     >>>
 
@@ -417,19 +431,24 @@ Storing the scraped data
 The simplest way to store the scraped data is by using :ref:`Feed exports
 <topics-feed-exports>`, with the following command::
 
-    scrapy crawl quotes -o items.json
+    scrapy crawl quotes -o quotes.json
 
-That will generate an ``items.json`` file containing all scraped items,
+That will generate an ``quotes.json`` file containing all scraped items,
 serialized in `JSON`_.
 
-You could've also used other formats, like `JSON Lines`_::
+For historic reasons, Scrapy appends to a given file instead of overwriting
+its contents. If you run this command twice without removing the file
+before the second time, you'll end up with a broken JSON file.
 
-    scrapy crawl quotes -o items.jl
+You can also used other formats, like `JSON Lines`_::
+
+    scrapy crawl quotes -o quotes.jl
 
 The `JSON Lines`_ format is useful because it's stream-like, you can easily
-append new records to it. As each record is a separate line, you can also
-process big files without having to fit everything in memory, there are tools
-like `JQ`_ to help doing that at the command-line.
+append new records to it. It doesn't have the same problem of JSON when you run
+twice. Also, as each record is a separate line, you can process big files
+without having to fit everything in memory, there are tools like `JQ`_ to help
+doing that at the command-line.
 
 In small projects (like the one in this tutorial), that should be enough.
 However, if you want to perform more complex things with the scraped items, you
@@ -466,14 +485,14 @@ markup:
 We can try extracting it in the shell::
 
     >>> response.css('li.next a').extract_first()
-    u'<a href="/page/2/">Next <span aria-hidden="true">\u2192</span></a>'
+    '<a href="/page/2/">Next <span aria-hidden="true">→</span></a>'
 
 This gets the anchor element, but we want the attribute ``href``. For that,
 Scrapy supports a CSS extension that let's you select the attribute contents,
 like this::
 
-    >>> response.css('li.next a::attr("href")').extract_first()
-    u'/page/2/'
+    >>> response.css('li.next a::attr(href)').extract_first()
+    '/page/2/'
 
 Let's see now our spider modified to recursively follow the link to the next
 page, extracting data from it::
@@ -495,7 +514,7 @@ page, extracting data from it::
                     'tags': quote.css("div.tags a.tag::text").extract(),
                 }
 
-            next_page = response.css('li.next a::attr("href")').extract_first()
+            next_page = response.css('li.next a::attr(href)').extract_first()
             if next_page is not None:
                 next_page = response.urljoin(next_page)
                 yield scrapy.Request(next_page, callback=self.parse)
@@ -540,12 +559,12 @@ this time for scraping author information::
 
         def parse(self, response):
             # follow links to author pages
-            for href in response.css('.author a::attr("href")').extract():
+            for href in response.css('.author a::attr(href)').extract():
                 yield scrapy.Request(response.urljoin(href),
                                      callback=self.parse_author)
 
             # follow pagination links
-            next_page = response.css('li.next a::attr("href")').extract_first()
+            next_page = response.css('li.next a::attr(href)').extract_first()
             if next_page is not None:
                 next_page = response.urljoin(next_page)
                 yield scrapy.Request(next_page, callback=self.parse)
@@ -586,7 +605,7 @@ Using spider arguments
 You can provide command line arguments to your spiders by using the ``-a``
 option when running them::
 
-    scrapy crawl quotes -o items.json -a tag=humor
+    scrapy crawl quotes -o quotes-humor.json -a tag=humor
 
 These arguments are passed to the Spider's ``__init__`` method and become
 spider attributes by default.  
@@ -606,7 +625,7 @@ with a specific tag, building the URL based on the argument::
             tag = getattr(self, 'tag', None)
             if tag is not None:
                 url = url + 'tag/' + tag
-            yield scrapy.Request(url)
+            yield scrapy.Request(url, self.parse)
 
         def parse(self, response):
             for quote in response.css('div.quote'):
@@ -615,10 +634,10 @@ with a specific tag, building the URL based on the argument::
                     'author': quote.css('span small a::text').extract_first(),
                 }
 
-            next_page = response.css('li.next a::attr("href")').extract_first()
+            next_page = response.css('li.next a::attr(href)').extract_first()
             if next_page is not None:
                 next_page = response.urljoin(next_page)
-                yield scrapy.Request(next_page, callback=self.parse)
+                yield scrapy.Request(next_page, self.parse)
 
 
 If you pass the ``tag=humor`` argument to this spider, you'll notice that it

From 38266cc949f594c7f596728876151a65b481c966 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Wed, 21 Sep 2016 11:02:24 -0300
Subject: [PATCH 17/20] recommend Dive into Python and Python tutorial instead
 of LPTHW for non-beginners

---
 docs/intro/tutorial.rst | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 65746c389..ec68bf922 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -20,14 +20,17 @@ This tutorial will walk you through these tasks:
 
 Scrapy is written in Python_. If you're new to the language you might want to
 start by getting an idea of what the language is like, to get the most out of
-Scrapy.  If you're already familiar with other languages, and want to learn
-Python quickly, we recommend `Learn Python The Hard Way`_.  If you're new to programming
-and want to start with Python, take a look at `this list of Python resources
-for non-programmers`_.
+Scrapy. If you're already familiar with other languages, and want to learn
+Python quickly, we recommend reading through `Dive Into Python 3`_.
+Alternatively, you can follow the `Python Tutorial`_. If you're new to
+programming and want to start with Python, take a look at `this list of Python
+resources for non-programmers`_.
 
 .. _Python: https://www.python.org/
 .. _this list of Python resources for non-programmers: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
-.. _Learn Python The Hard Way: http://learnpythonthehardway.org/book/
+.. _Dive Into Python 3: http://www.diveintopython3.net
+.. _Python Tutorial: https://docs.python.org/3/tutorial
+
 
 Creating a project
 ==================

From 32017a76f8560d8cba2746d541adb63e03e68f62 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Wed, 21 Sep 2016 11:06:36 -0300
Subject: [PATCH 18/20] recommend learn python the hard way for beginners

---
 docs/intro/tutorial.rst | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index ec68bf922..31228017b 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -20,16 +20,21 @@ This tutorial will walk you through these tasks:
 
 Scrapy is written in Python_. If you're new to the language you might want to
 start by getting an idea of what the language is like, to get the most out of
-Scrapy. If you're already familiar with other languages, and want to learn
-Python quickly, we recommend reading through `Dive Into Python 3`_.
-Alternatively, you can follow the `Python Tutorial`_. If you're new to
-programming and want to start with Python, take a look at `this list of Python
-resources for non-programmers`_.
+Scrapy.
+
+If you're already familiar with other languages, and want to learn Python
+quickly, we recommend reading through `Dive Into Python 3`_.  Alternatively,
+you can follow the `Python Tutorial`_.
+
+If you're new to programming and want to start with Python, you may find useful
+the online book `Learn Python The Hard Way`_. You can also take a look at `this
+list of Python resources for non-programmers`_.
 
 .. _Python: https://www.python.org/
 .. _this list of Python resources for non-programmers: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
 .. _Dive Into Python 3: http://www.diveintopython3.net
 .. _Python Tutorial: https://docs.python.org/3/tutorial
+.. _Learn Python The Hard Way: http://learnpythonthehardway.org/book/
 
 
 Creating a project

From d636e5baa8a077e2869bfe3b76525efec42392ec Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Wed, 21 Sep 2016 18:54:12 -0300
Subject: [PATCH 19/20] better description for start_requests expected return
 value

---
 docs/intro/tutorial.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 31228017b..e85219e06 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -104,9 +104,10 @@ and defines some attributes and methods:
   unique within a project, that is, you can't set the same name for different
   Spiders.
 
-* :meth:`~scrapy.spiders.Spider.start_requests`: must generate or return a list
-  of requests where the Spider will begin to crawl from.
-  Subsequent requests will be generated successively from these initial requests.
+* :meth:`~scrapy.spiders.Spider.start_requests`: must return an iterable of
+  Requests (you can return a list of requests or write a generator function)
+  which the Spider will begin to crawl from. Subsequent requests will be
+  generated successively from these initial requests.
 
 * :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
   the response downloaded for each of the requests made. The response parameter

From f4a22089168c31a2b6c2f03c0053073eb80e33b3 Mon Sep 17 00:00:00 2001
From: Elias Dorneles <eliasdorneles@gmail.com>
Date: Thu, 22 Sep 2016 11:04:45 -0300
Subject: [PATCH 20/20] addressing review comments and other minor editing

---
 docs/intro/tutorial.rst | 63 +++++++++++++++++++++--------------------
 1 file changed, 33 insertions(+), 30 deletions(-)

diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index e85219e06..4f2736709 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -111,8 +111,8 @@ and defines some attributes and methods:
 
 * :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle
   the response downloaded for each of the requests made. The response parameter
-  is an instance of :class:`~scrapy.http.Response` that holds the page content and
-  has further helpful methods to handle it.
+  is an instance of :class:`~scrapy.http.TextResponse` that holds
+  the page content and has further helpful methods to handle it.
 
   The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting
   the scraped data as dicts and also finding new URLs to
@@ -154,7 +154,7 @@ What just happened under the hood?
 
 Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects
 returned by the ``start_requests`` method of the Spider. Upon receiving a
-response for each one, it instantiates :class:`scrapy.http.Response` objects
+response for each one, it instantiates :class:`~scrapy.http.Response` objects
 and calls the callback method associated with the request (in this case, the
 ``parse`` method) passing the response as argument.
 
@@ -281,11 +281,11 @@ expressions`::
 
 In order to find the proper CSS selectors to use, you might find useful opening
 the response page from the shell in your web browser using ``view(response)``.
-You can use your browser developer tools or extensions like Firebug.  For more
-information see :ref:`topics-firebug` and :ref:`topics-firefox`.
+You can use your browser developer tools or extensions like Firebug (see
+sections about :ref:`topics-firebug` and :ref:`topics-firefox`).
 
 `Selector Gadget`_ is also a nice tool to quickly find CSS selector for
-visually selected elements.
+visually selected elements, which works in many browsers.
 
 .. _regular expressions: https://docs.python.org/3/library/re.html
 .. _Selector Gadget: http://selectorgadget.com/
@@ -308,13 +308,13 @@ objects in the shell.
 
 While perhaps not as popular as CSS selectors, XPath expressions offer more
 power because besides navigating the structure, it can also look at the
-content. Using XPath, you're able to select things like: **select the link
-that contains the text "Next Page"**. This makes XPath very fitting to the task
+content. Using XPath, you're able to select things like: *select the link
+that contains the text "Next Page"*. This makes XPath very fitting to the task
 of scraping, and we encourage you to learn XPath even if you already know how to
 construct CSS selectors, it will make scraping much easier.
 
-We won't cover much of XPath here, but you can read more about `using XPath
-with Scrapy Selectors here <topics-selectors>`_. To learn more about XPath, we
+We won't cover much of XPath here, but you can read more about :ref:`using XPath
+with Scrapy Selectors here <topics-selectors>`. To learn more about XPath, we
 recommend `this tutorial to learn XPath through examples
 <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this tutorial to learn "how
 to think in XPath" <http://plasmasturm.org/log/xpath101/>`_.
@@ -352,7 +352,7 @@ like this:
 Let's open up scrapy shell and play a bit to find out how to extract the data
 we want::
 
-    $ scrapy shell http://quotes.toscrape.com
+    $ scrapy shell 'http://quotes.toscrape.com'
 
 We get a list of selectors for the quote HTML elements with::
 
@@ -394,7 +394,6 @@ quotes elements and put them together into a Python dictionary::
         ... a few more of these, omitted for brevity
     >>>
 
-
 Extracting data in our spider
 ------------------------------
 
@@ -421,7 +420,7 @@ in the callback, as you can see below::
                 yield {
                     'text': quote.css('span.text::text').extract_first(),
                     'author': quote.css('span small::text').extract_first(),
-                    'tags': quote.css("div.tags a.tag::text").extract(),
+                    'tags': quote.css('div.tags a.tag::text').extract(),
                 }
 
 If you run this spider, it will output the extracted data with the log::
@@ -520,7 +519,7 @@ page, extracting data from it::
                 yield {
                     'text': quote.css('span.text::text').extract_first(),
                     'author': quote.css('span small::text').extract_first(),
-                    'tags': quote.css("div.tags a.tag::text").extract(),
+                    'tags': quote.css('div.tags a.tag::text').extract(),
                 }
 
             next_page = response.css('li.next a::attr(href)').extract_first()
@@ -530,10 +529,11 @@ page, extracting data from it::
 
 
 Now, after extracting the data, the ``parse()`` method looks for the link to
-the next page, builds a full absolute URL using the ``response.urljoin`` method
-(since the links can be relative) and yields a new request to the next page,
-registering itself as callback to handle the data extraction for the next page
-and to keep the crawling going through all the pages.
+the next page, builds a full absolute URL using the
+:meth:`~scrapy.http.Response.urljoin` method (since the links can be
+relative) and yields a new request to the next page, registering itself as
+callback to handle the data extraction for the next page and to keep the
+crawling going through all the pages.
 
 What you see here is Scrapy's mechanism of following links: when you yield
 a Request in a callback method, Scrapy will schedule that request to be sent
@@ -547,12 +547,8 @@ In our example, it creates a sort of loop, following all the links to the next p
 until it doesn't find one -- handy for crawling blogs, forums and other sites with
 pagination.
 
-Another common pattern is to build an item with data from more than one page,
-using a :ref:`trick to pass additional data to the callbacks
-<topics-request-response-ref-request-callback-arguments>`.
-
-Another example: scraping authors
----------------------------------
+More examples and patterns
+--------------------------
 
 Here is another spider that illustrates callbacks and following links,
 this time for scraping author information::
@@ -602,11 +598,18 @@ requests to URLs already visited, avoiding the problem of hitting servers too
 much because of a programming mistake. This can be configured by the setting
 :setting:`DUPEFILTER_CLASS`.
 
-.. note::
-    As another example spider that leverages the mechanism of following links,
-    check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic
-    spider that implements a small rules engine that you can use to write your
-    crawlers on top of it.
+Hopefully by now you have a good understanding of how to use the mechanism
+of following links and callbacks with Scrapy.
+
+As yet another example spider that leverages the mechanism of following links,
+check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic
+spider that implements a small rules engine that you can use to write your
+crawlers on top of it.
+
+Also, a common pattern is to build an item with data from more than one page,
+using a :ref:`trick to pass additional data to the callbacks
+<topics-request-response-ref-request-callback-arguments>`.
+
 
 Using spider arguments
 ======================
@@ -663,7 +666,7 @@ features not mentioned here. Check the :ref:`topics-whatelse` section in
 :ref:`intro-overview` chapter for a quick overview of the most important ones.
 
 You can continue from the section :ref:`section-basics` to know more about the
-command-line tool, spiders and other things the tutorial haven't covered like
+command-line tool, spiders, selectors and other things the tutorial hasn't covered like
 modeling the scraped data. If you prefer to play with an example project, check
 the :ref:`intro-examples` section.