From 76e3bf12508f51c8c7b532b1a55880e55e144f19 Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Thu, 26 Mar 2015 14:26:20 -0300 Subject: [PATCH] addressing comments from the review plus further editing --- docs/intro/overview.rst | 83 +++++++++++++++++++++++------------- docs/topics/autothrottle.rst | 2 + 2 files changed, 56 insertions(+), 29 deletions(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 295208bd2..d93797cae 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -39,15 +39,11 @@ voted questions on StackOverflow and scrapes some data from each page:: yield scrapy.Request(full_url, callback=self.parse_question) def parse_question(self, response): - title = response.css('h1 a::text').extract_first() - votes = response.css('.question .vote-count-post::text').extract_first() - tags = response.css('.question .post-tag::text').extract() - body = response.css('.question .post-text').extract_first() yield { - 'title': title, - 'votes': votes, - 'body': body, - 'tags': tags, + 'title': response.css('h1 a::text').extract()[0], + 'votes': response.css('.question .vote-count-post::text').extract()[0], + 'body': response.css('.question .post-text').extract()[0], + 'tags': response.css('.question .post-tag::text').extract(), 'link': response.url, } @@ -66,28 +62,36 @@ title, link, number of upvotes, a list of the tags and the question content in H What just happened? ------------------- -When you ran the command ``scrapy runspider somefile.py``, Scrapy looked -for a Spider definition inside it and ran it through its crawler engine. +When you ran the command ``scrapy runspider somefile.py``, Scrapy looked for a +Spider definition inside it and ran it through its crawler engine. The crawl started by making requests to the URLs defined in the ``start_urls`` attribute (in this case, only the URL for StackOverflow top questions page), -and then called the default callback method ``parse`` passing the response -object as an argument. +and called the default callback method ``parse`` passing the response object as +an argument. In the ``parse`` callback, we extract the links to the +question pages using a CSS Selector with a custom extension that allows to get +the value for an attribute. Then, we yield a few more requests to be sent, +registering the method ``parse_question`` as the callback to be called for each +of them as they finish. Here you notice one of the main advantages about Scrapy: requests are -scheduled and processed asynchronously. This means that Scrapy doesn't -need to wait for a request to be finished and processed, it can send -another request or do other things in the meantime, which results in much -faster crawlings. +:ref:`scheduled and processed asynchronously `. This +means that Scrapy doesn't need to wait for a request to be finished and +processed, it can send another request or do other things in the meantime. This +also means that other requests can keep going even if some request fails or an +error happens while handling it. -So, in the ``parse`` callback, we scrape the links to the questions and -yield a few more requests to be done, registering for them the method -``parse_question`` as the callback to be called for each of them as -they finish. +While this enables you to do very fast crawlings sending multiple concurrent +requests at the same time in a fault-tolerant way, Scrapy also gives you +control over the politeness of the crawl through :ref:`a few settings +`. You can do things like setting a download delay between +each request, limit amount of concurrent requests per domain or per IP, and +even :ref:`use an auto-throttling extension ` that tries +to figure out these automatically. -Finally, the ``parse_question`` callback scrapes the question data -for each page yielding a dict, which Scrapy then collects and -writes to a JSON file as requested in the command line. +Finally, the ``parse_question`` callback scrapes the question data for each +page yielding a dict, which Scrapy then collects and writes to a JSON file as +requested in the command line. .. note:: @@ -96,6 +100,25 @@ writes to a JSON file as requested in the command line. storage backend (FTP or `Amazon S3`_, for example). You can also write an :ref:`item pipeline ` to store the items in a database. +The data in the file will look like this (note: formatted for easier reading):: + + + [{ + "body": "... LONG HTML HERE ...", + "link": "http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array", + "tags": ["java", "c++", "performance", "optimization"], + "title": "Why is processing a sorted array faster than an unsorted array?", + "votes": "9924" + }, + { + "body": "... LONG HTML HERE ...", + "link": "http://stackoverflow.com/questions/1260748/how-do-i-remove-a-git-submodule", + "tags": ["git", "git-submodules"], + "title": "How do I remove a Git submodule?", + "votes": "1764" + }, + ...] + .. _topics-whatelse: @@ -106,6 +129,10 @@ You've seen how to extract and store items from a website using Scrapy, but this is just the surface. Scrapy provides a lot of powerful features for making scraping easy and efficient, such as: +* Built-in support for :ref:`selecting and extracting ` data + from HTML/XML sources using CSS selectors extended and XPath expressions, + with helper methods to extract using regular expressions. + * An :ref:`interactive shell console ` (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when writing or debugging your spiders. @@ -126,12 +153,10 @@ scraping easy and efficient, such as: console running inside your Scrapy process, to introspect and debug your crawler -* A caching DNS resolver - -* Support for crawling based on URLs discovered through `Sitemaps`_ - -* A media pipeline for :ref:`automatically downloading images ` - (or any other media) associated with the scraped items +* Plus other goodies like reusable spiders to crawl sites from `Sitemaps`_ and + XML/CSV feeds, a media pipeline for :ref:`automatically downloading images ` + (or any other media) associated with the scraped items, a caching DNS resolver, + and much more! What's next? ============ diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rst index 475946a54..8073ec6e0 100644 --- a/docs/topics/autothrottle.rst +++ b/docs/topics/autothrottle.rst @@ -1,3 +1,5 @@ +.. _topics-autothrottle: + ====================== AutoThrottle extension ======================