mirror of https://github.com/scrapy/scrapy.git
docs: update overview spider code to use toscrape.com and minor changes
So, this will replace the spider example code from the overview that scrapes questions from StackOverflow by a spider scraping quotes (much like the one in the tutorial), and upates the text around it to be consistent. There are also minor wording changes plus a small Sphinx/reST syntax fix on the features list at the bottom (it was creating a definition list, causing one line to be bold).
This commit is contained in:
Elias Dorneles
parent
2f60f2a5a6
commit
18bd0b0886
|
|
@ -19,74 +19,69 @@ Walk-through of an example spider
|
|||
In order to show you what Scrapy brings to the table, we'll walk you through an
|
||||
example of a Scrapy Spider using the simplest way to run a spider.
|
||||
|
||||
So, here's the code for a spider that follows the links to the top
|
||||
voted questions on StackOverflow and scrapes some data from each page::
|
||||
Here's the code for a spider that scrapes famous quotes from website
|
||||
http://quotes.toscrape.com, following the pagination::
|
||||
|
||||
import scrapy
|
||||
|
||||
|
||||
class StackOverflowSpider(scrapy.Spider):
|
||||
name = 'stackoverflow'
|
||||
start_urls = ['http://stackoverflow.com/questions?sort=votes']
|
||||
class QuotesSpider(scrapy.Spider):
|
||||
name = "quotes"
|
||||
start_urls = [
|
||||
'http://quotes.toscrape.com/tag/humor/',
|
||||
]
|
||||
|
||||
def parse(self, response):
|
||||
for href in response.css('.question-summary h3 a::attr(href)'):
|
||||
full_url = response.urljoin(href.extract())
|
||||
yield scrapy.Request(full_url, callback=self.parse_question)
|
||||
for quote in response.css('div.quote'):
|
||||
yield {
|
||||
'text': quote.css('span.text::text').extract_first(),
|
||||
'author': quote.xpath('span/small/text()').extract_first(),
|
||||
}
|
||||
|
||||
def parse_question(self, response):
|
||||
yield {
|
||||
'title': response.css('h1 a::text').extract_first(),
|
||||
'votes': response.css('.question .vote-count-post::text').extract_first(),
|
||||
'body': response.css('.question .post-text').extract_first(),
|
||||
'tags': response.css('.question .post-tag::text').extract(),
|
||||
'link': response.url,
|
||||
}
|
||||
next_page = response.css('li.next a::attr("href")').extract_first()
|
||||
if next_page:
|
||||
next_page = response.urljoin(next_page)
|
||||
yield scrapy.Request(next_page, callback=self.parse)
|
||||
|
||||
|
||||
Put this in a file, name it to something like ``stackoverflow_spider.py``
|
||||
Put this in a text file, name it to something like ``quotes_spider.py``
|
||||
and run the spider using the :command:`runspider` command::
|
||||
|
||||
scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json
|
||||
scrapy runspider quotes_spider.py -o quotes.json
|
||||
|
||||
|
||||
When this finishes you will have in the ``top-stackoverflow-questions.json`` file
|
||||
a list of the most upvoted questions in StackOverflow in JSON format, containing the
|
||||
title, link, number of upvotes, a list of the tags and the question content in HTML,
|
||||
looking like this (reformatted for easier reading)::
|
||||
When this finishes you will have in the ``quotes.json`` file a list of the
|
||||
quotes in JSON format, containing text and author, looking like this (reformatted
|
||||
here for better readability)::
|
||||
|
||||
[{
|
||||
"body": "... LONG HTML HERE ...",
|
||||
"link": "http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array",
|
||||
"tags": ["java", "c++", "performance", "optimization"],
|
||||
"title": "Why is processing a sorted array faster than an unsorted array?",
|
||||
"votes": "9924"
|
||||
"author": "Jane Austen",
|
||||
"text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
|
||||
},
|
||||
{
|
||||
"body": "... LONG HTML HERE ...",
|
||||
"link": "http://stackoverflow.com/questions/1260748/how-do-i-remove-a-git-submodule",
|
||||
"tags": ["git", "git-submodules"],
|
||||
"title": "How do I remove a Git submodule?",
|
||||
"votes": "1764"
|
||||
"author": "Groucho Marx",
|
||||
"text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
|
||||
},
|
||||
{
|
||||
"author": "Steve Martin",
|
||||
"text": "\u201cA day without sunshine is like, you know, night.\u201d"
|
||||
},
|
||||
...]
|
||||
|
||||
|
||||
|
||||
What just happened?
|
||||
-------------------
|
||||
|
||||
When you ran the command ``scrapy runspider somefile.py``, Scrapy looked for a
|
||||
When you ran the command ``scrapy runspider quotes_spider.py``, Scrapy looked for a
|
||||
Spider definition inside it and ran it through its crawler engine.
|
||||
|
||||
The crawl started by making requests to the URLs defined in the ``start_urls``
|
||||
attribute (in this case, only the URL for StackOverflow top questions page)
|
||||
attribute (in this case, only the URL for quotes in *humor* category)
|
||||
and called the default callback method ``parse``, passing the response object as
|
||||
an argument. In the ``parse`` callback we extract the links to the
|
||||
question pages using a CSS Selector with a custom extension that allows to get
|
||||
the value for an attribute. Then we yield a few more requests to be sent,
|
||||
registering the method ``parse_question`` as the callback to be called for each
|
||||
of them as they finish.
|
||||
an argument. In the ``parse`` callback we loop through the quote elements
|
||||
using a CSS Selector, yield a Python dict with the extracted quote text and author,
|
||||
look for a link to the next page and schedules another request using the same
|
||||
``parse`` method as callback.
|
||||
|
||||
Here you notice one of the main advantages about Scrapy: requests are
|
||||
:ref:`scheduled and processed asynchronously <topics-architecture>`. This
|
||||
|
|
@ -103,10 +98,6 @@ each request, limiting amount of concurrent requests per domain or per IP, and
|
|||
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
|
||||
to figure out these automatically.
|
||||
|
||||
Finally, the ``parse_question`` callback scrapes the question data for each
|
||||
page yielding a dict, which Scrapy then collects and writes to a JSON file as
|
||||
requested in the command line.
|
||||
|
||||
.. note::
|
||||
|
||||
This is using :ref:`feed exports <topics-feed-exports>` to generate the
|
||||
|
|
@ -145,12 +136,13 @@ scraping easy and efficient, such as:
|
|||
:ref:`pipelines <topics-item-pipeline>`).
|
||||
|
||||
* Wide range of built-in extensions and middlewares for handling:
|
||||
* cookies and session handling
|
||||
* HTTP features like compression, authentication, caching
|
||||
* user-agent spoofing
|
||||
* robots.txt
|
||||
* crawl depth restriction
|
||||
* and more
|
||||
|
||||
- cookies and session handling
|
||||
- HTTP features like compression, authentication, caching
|
||||
- user-agent spoofing
|
||||
- robots.txt
|
||||
- crawl depth restriction
|
||||
- and more
|
||||
|
||||
* A :ref:`Telnet console <topics-telnetconsole>` for hooking into a Python
|
||||
console running inside your Scrapy process, to introspect and debug your
|
||||
|
|
@ -165,8 +157,8 @@ What's next?
|
|||
============
|
||||
|
||||
The next steps for you are to :ref:`install Scrapy <intro-install>`,
|
||||
:ref:`follow through the tutorial <intro-tutorial>` to learn how to organize
|
||||
your code in Scrapy projects and `join the community`_. Thanks for your
|
||||
:ref:`follow through the tutorial <intro-tutorial>` to learn how to create
|
||||
a full-blown Scrapy project and `join the community`_. Thanks for your
|
||||
interest!
|
||||
|
||||
.. _join the community: http://scrapy.org/community/
|
||||
|
|
|
|||
Loading…
Reference in New Issue