RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/intro/tutorial.rst

19 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy Tutorial

In this tutorial, we'll assume that Scrapy is already installed on your system. If that's not the case, see :ref:`intro-install`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

We are going to scrape quotes.toscrape.com, a website that lists quotes from famous authors.

This tutorial will walk you through these tasks:

  1. Creating a new Scrapy project

  2. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract data

    System Message: ERROR/3 (<stdin>, line 16); backlink

    Unknown interpreted text role "ref".

  3. Exporting the scraped data using command line

Scrapy is written in Python. If you're new to the language you might want to start by getting an idea of what the language is like, to get the most out of Scrapy. If you're already familiar with other languages, and want to learn Python quickly, we recommend Learn Python The Hard Way. If you're new to programming and want to start with Python, take a look at this list of Python resources for non-programmers.

Creating a project

Before you start scraping, you will have to set up a new Scrapy project. Enter a directory where you'd like to store your code and run:

scrapy startproject tutorial

This will create a tutorial directory with the following contents:

tutorial/
    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here
        __init__.py

        items.py          # project items file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders
            __init__.py

Our first Spider

Spiders are classes that you define and that Scrapy uses to scrape information from a website (or group of websites). They define the initial requests to make, how to follow links in the pages, and how to parse the downloaded page content to extract data.

This is the code for our first Spider. Save it in a file named quotes_spider.py under the tutorial/spiders directory in your project:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)

As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.spiders.Spider>` and defines some attributes and methods:

System Message: ERROR/3 (<stdin>, line 88); backlink

Unknown interpreted text role "class".

  • :attr:`~scrapy.spiders.Spider.name`: identifies the Spider. It must be unique within a project, that is, you can't set the same name for different Spiders.

    System Message: ERROR/3 (<stdin>, line 91); backlink

    Unknown interpreted text role "attr".

  • :meth:`~scrapy.spiders.Spider.start_requests`: must return a list of requests where the Spider will begin to crawl from. Subsequent requests will be generated successively from these initial requests.

    System Message: ERROR/3 (<stdin>, line 95); backlink

    Unknown interpreted text role "meth".

  • :meth:`~scrapy.spiders.Spider.parse`: a method that will be called to handle the response downloaded for each of the requests made. The response parameter is an instance of :class:`~scrapy.http.Response` that holds the page content and has further helpful methods to handle it.

    System Message: ERROR/3 (<stdin>, line 99); backlink

    Unknown interpreted text role "meth".

    System Message: ERROR/3 (<stdin>, line 99); backlink

    Unknown interpreted text role "class".

    The :meth:`~scrapy.spiders.Spider.parse` method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new requests (:class:`~scrapy.http.Request`) from them.

    System Message: ERROR/3 (<stdin>, line 104); backlink

    Unknown interpreted text role "meth".

    System Message: ERROR/3 (<stdin>, line 104); backlink

    Unknown interpreted text role "class".

How to run your spider

To put our spider to work, go to the project's top level directory and run:

scrapy crawl quotes

This command runs the spider with name quotes that we've just added, that will send some requests for the quotes.toscrape.com domain. You will get an output similar to this:

2016-09-01 16:51:27 [scrapy] INFO: Scrapy started (bot: tutorial)
2016-09-01 16:51:27 [scrapy] INFO: Overridden settings: {...}
2016-09-01 16:51:27 [scrapy] INFO: Enabled extensions: ...
2016-09-01 16:51:27 [scrapy] INFO: Enabled downloader middlewares: ...
2016-09-01 16:51:27 [scrapy] INFO: Enabled spider middlewares: ...
2016-09-01 16:51:27 [scrapy] INFO: Enabled item pipelines: ...
2016-09-01 16:51:27 [scrapy] INFO: Spider opened
2016-09-01 16:51:27 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2016-09-01 16:51:28 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
2016-09-01 16:51:29 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
2016-09-01 16:51:29 [scrapy] INFO: Closing spider (finished)

Now, check the files in the current directory. You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs.

Note

If you are wondering why we haven't parsed the HTML yet, hold on, we will cover that soon.

What just happened under the hood?

Scrapy schedules the :class:`scrapy.Request <scrapy.http.Request>` objects returned by the start_requests method of the Spider. Upon receiving a response for each one, it instantiates :class:`scrapy.http.Response` objects and calls the parse callback method passing the response as argument.

System Message: ERROR/3 (<stdin>, line 143); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 143); backlink

Unknown interpreted text role "class".

Simplifying your spider

Instead of defining the :meth:`~scrapy.spiders.Spider.start_requests` method generating :class:`scrapy.Request <scrapy.http.Request>` objects from URLs, you can just put those URLs in the :attr:`~scrapy.spiders.Spider.start_urls` attribute:

System Message: ERROR/3 (<stdin>, line 152); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 152); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 152); backlink

Unknown interpreted text role "attr".
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
      'http://quotes.toscrape.com/page/1/',
      'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)

The :meth:`~scrapy.spiders.Spider.parse` method will be called to handle each of the requests for those URLs, even though we haven't explicitely told Scrapy to do so. This happens because :meth:`~scrapy.spiders.Spider.parse` is Scrapy's default callback method.

System Message: ERROR/3 (<stdin>, line 173); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 173); backlink

Unknown interpreted text role "meth".

Extracting Items

Introduction to Selectors

There are several ways to extract data from web pages. Scrapy uses a mechanism based on XPath or CSS expressions called :ref:`Scrapy Selectors <topics-selectors>`. For more information about selectors and other extraction mechanisms see the :ref:`Selectors documentation <topics-selectors>`.

System Message: ERROR/3 (<stdin>, line 185); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 185); backlink

Unknown interpreted text role "ref".

Here are some examples of XPath expressions and their meanings:

  • /html/head/title: selects the <title> element, inside the <head> element of an HTML document. Equivalent CSS selector: html > head > title.
  • /html/head/title/text(): selects the text inside the aforementioned <title> element. Equivalent CSS selector: html > head > title ::text.
  • //td: selects all the <td> elements from the whole document. Equivalent CSS selector: td.
  • //div[@class="mine"]: selects all div elements which contain an attribute class="mine". Equivalent CSS selector: div.mine.

These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much more powerful. To learn more about XPath, we recommend this tutorial to learn XPath through examples, and this tutorial to learn "how to think in XPath".

Note

CSS vs XPath: you can go a long way extracting data from web pages using only CSS selectors. However, XPath offers more power because besides navigating the structure, it can also look at the content: you're able to select things like: the link that contains the text 'Next Page'. Because of this, we encourage you to learn about XPath even if you already know how to construct CSS selectors.

For working with CSS and XPath expressions, Scrapy provides the :class:`~scrapy.selector.Selector` class and convenient shortcuts to avoid instantiating selectors yourself every time you need to select something from a response.

System Message: ERROR/3 (<stdin>, line 220); backlink

Unknown interpreted text role "class".

You can see selectors as objects that represent nodes in the document structure. So, the first instantiated selectors are associated with the root node, or the entire document.

Selectors have four basic methods (click on the method to see the complete API documentation):

Trying Selectors in the Shell

To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy shell <topics-shell>`, which also requires IPython (an extended Python console) installed on your system.

System Message: ERROR/3 (<stdin>, line 249); backlink

Unknown interpreted text role "ref".

To start a shell, you must go to the project's top level directory and run:

scrapy shell "http://quotes.toscrape.com"

Note

Remember to always enclose urls in quotes when running Scrapy shell from command-line, otherwise urls containing arguments (ie. & character) will not work.

This is what the shell looks like:

[ ... Scrapy log here ... ]

2016-09-01 18:14:39 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x109001c90>
[s]   item       {}
[s]   request    <GET http://quotes.toscrape.com>
[s]   response   <200 http://quotes.toscrape.com>
[s]   settings   <scrapy.settings.Settings object at 0x109001610>
[s]   spider     <DefaultSpider 'default' at 0x1092808d0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

>>>

After the shell loads, you will have the response fetched in a local response variable, so if you type response.body you will see the body of the response, or you can type response.headers to see its headers.

More importantly response has a selector attribute which is an instance of :class:`~scrapy.selector.Selector` class, instantiated with this particular response. You can run queries on response by calling response.selector.xpath() or response.selector.css(). There are also some convenience shortcuts like response.xpath() or response.css() which map directly to response.selector.xpath() and response.selector.css().

System Message: ERROR/3 (<stdin>, line 286); backlink

Unknown interpreted text role "class".

So let's try it:

In [1]: response.xpath('//title')
Out[1]: [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]

In [2]: response.xpath('//title').extract()
Out[2]: [u'<title>Quotes to Scrape</title>']

In [3]: response.xpath('//title/text()')
Out[3]: [<Selector xpath='//title/text()' data=u'Quotes to Scrape'>]

In [4]: response.xpath('//title/text()').extract()
Out[4]: [u'Quotes to Scrape']

In [11]: response.xpath('//title/text()').re('(\w+)')
Out[11]: [u'Quotes', u'to', u'Scrape']

Extracting the data

Now, let's try to extract some real information from those pages.

You could type response.body in the console, and inspect the source code to figure out the XPaths you need to use. However, inspecting the raw HTML code there could become a very tedious task. To make it easier, you can use Firefox Developer Tools or some Firefox extensions like Firebug. For more information see :ref:`topics-firebug` and :ref:`topics-firefox`.

System Message: ERROR/3 (<stdin>, line 316); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 316); backlink

Unknown interpreted text role "ref".

After inspecting the page source, you'll find that every quote in the website is inside a separate <div class="quote"> element, such as:

<div class="quote">
    <span class="text">“We accept the love we think we deserve.”</span>
    <span>by <small class="author">Stephen Chbosky</small></span>
    <div class="tags">
        Tags:
        <meta class="keywords">
        <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
        <a class="tag" href="/tag/love/page/1/">love</a>
    </div>
</div>

So we can select each <div class="quote"> element belonging to the site's list with this code:

response.xpath('//div[@class="quote"]')

From the quote elements, we can select the texts with:

response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()

The authors:

response.xpath('//div[@class="quote"]/span/small/text()').extract()

As we've said before, each .xpath() call returns a list of selectors, so we can concatenate further .xpath() calls to dig deeper into a node. We are going to use that property here, so:

for quote in response.xpath('//div[@class="quote"]'):
    text = quote.xpath('span[@class="text"]/text()').extract_first()
    author = quote.xpath('span/small/text()').extract_first()
    print({'text': text, 'author': author})

In the above snippet we've decided to use the method .extract_first() instead of .extract(), to extract the content from the first element from a selector list returned by .xpath().

Note

For a more detailed description of using nested selectors, see :ref:`topics-selectors-nesting-selectors` and :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors` documentation

System Message: ERROR/3 (<stdin>, line 365); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 365); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 365); backlink

Unknown interpreted text role "ref".

Knowing to use selectors, extracting data from a page is just a matter of yield the Python dictionaries from the callback method instead of printing them.

Let's add the necessary code to our spider:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('span/small/text()').extract_first(),
            }

Run:

scrapy crawl quotes

Now crawling quotes.toscrape.com will show dictionary objects:

2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'author': 'Oscar Wilde',
 'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'author': 'Mark Twain',
 'text': '“The man who does not read has no advantage over the man who cannot read.”'}

Following links

Let's say, instead of just scraping the stuff from the first two pages from quotes.toscrape.com, you want quotes from all the pages in the website.

Now that you know how to extract data from a page, why not extract the pagination links in each page, follow them and then extract the data you want for all of them?

Here is a modification to our spider that does just that:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('span/small/text()').extract_first(),
            }

        next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

Now after extracting an item the parse() method looks for the link to the next page, builds a full absolute URL using the response.urljoin method (since the links can be relative) and yields a new request to the next page, registering itself as callback to handle the data extraction for the next page and to keep the crawling going through all the pages.

What you see here is Scrapy's mechanism of following links: when you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes.

Using this, you can build complex crawlers that follow links according to rules you define, and extract different kinds of data depending on the page it's visiting.

In our example, it creates a sort of loop, following all the links to the next page until it doesn't find one -- handy for crawling blogs, forums and other sites with pagination.

Another common pattern is to build an item with data from more than one page, using a :ref:`trick to pass additional data to the callbacks <topics-request-response-ref-request-callback-arguments>`.

System Message: ERROR/3 (<stdin>, line 456); backlink

Unknown interpreted text role "ref".

Note

As an example spider that leverages this mechanism, check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic spider that implements a small rules engine that you can use to write your crawlers on top of it.

System Message: ERROR/3 (<stdin>, line 462); backlink

Unknown interpreted text role "class".

Storing the scraped data

The simplest way to store the scraped data is by using :ref:`Feed exports <topics-feed-exports>`, with the following command:

System Message: ERROR/3 (<stdin>, line 470); backlink

Unknown interpreted text role "ref".
scrapy crawl quotes -o items.json

That will generate an items.json file containing all scraped items, serialized in JSON.

In small projects (like the one in this tutorial), that should be enough. However, if you want to perform more complex things with the scraped items, you can write an :ref:`Item Pipeline <topics-item-pipeline>`. As with Items, a placeholder file for Item Pipelines has been set up for you when the project is created, in tutorial/pipelines.py. Though you don't need to implement any item pipelines if you just want to store the scraped items.

System Message: ERROR/3 (<stdin>, line 478); backlink

Unknown interpreted text role "ref".

Next steps

This tutorial covered only the basics of Scrapy, but there's a lot of other features not mentioned here. Check the :ref:`topics-whatelse` section in :ref:`intro-overview` chapter for a quick overview of the most important ones.

System Message: ERROR/3 (<stdin>, line 488); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 488); backlink

Unknown interpreted text role "ref".

Then, we recommend you continue by playing with an example project (see :ref:`intro-examples`), and then continue with the section :ref:`section-basics`.

System Message: ERROR/3 (<stdin>, line 492); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 492); backlink

Unknown interpreted text role "ref".
</html>