RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/intro/overview.rst

8.9 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy at a glance

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for screen scraping (more precisely, web scraping), it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

The purpose of this document is to introduce you to the concepts behind Scrapy so you can get an idea of how it works and decide if Scrapy is what you need.

When you're ready to start a project, you can :ref:`start with the tutorial <intro-tutorial>`.

System Message: ERROR/3 (<stdin>, line 19); backlink

Unknown interpreted text role "ref".

Pick a website

So you need to extract some information from a website, but the website doesn't provide any API or mechanism to access that info programmatically. Scrapy can help you extract that information.

Let's say we want to extract the URL, name, description and size of all torrent files added today in the Mininova site.

The list of all torrents added today can be found on this page:

http://www.mininova.org/today

Define the data you want to scrape

The first thing is to define the data we want to scrape. In Scrapy, this is done through :ref:`Scrapy Items <topics-items>` (Torrent files, in this case).

System Message: ERROR/3 (<stdin>, line 41); backlink

Unknown interpreted text role "ref".

This would be our Item:

import scrapy

class TorrentItem(scrapy.Item):
    url = scrapy.Field()
    name = scrapy.Field()
    description = scrapy.Field()
    size = scrapy.Field()

Write a Spider to extract the data

The next thing is to write a Spider which defines the start URL (http://www.mininova.org/today), the rules for following links and the rules for extracting the data from pages.

If we take a look at that page content we'll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We'll use that to construct the regular expression for the links to follow: /tor/\d+.

We'll use XPath for selecting the data to extract from the web page HTML source. Let's take one of those torrent pages:

http://www.mininova.org/tor/2676093

And look at the page HTML source to construct the XPath to select the data we want which is: torrent name, description and size.

System Message: ERROR/3 (<stdin>, line 73)

Unknown directive type "highlight".

.. highlight:: html

By looking at the page HTML source we can see that the file name is contained inside a <h1> tag:

<h1>Darwin - The Evolution Of An Exhibition</h1>

System Message: ERROR/3 (<stdin>, line 80)

Unknown directive type "highlight".

.. highlight:: none

An XPath expression to extract the name could be:

//h1/text()

System Message: ERROR/3 (<stdin>, line 86)

Unknown directive type "highlight".

.. highlight:: html

And the description is contained inside a <div> tag with id="description":

<h2>Description:</h2>

<div id="description">
Short documentary made for Plymouth City Museum and Art Gallery regarding the setup of an exhibit about Charles Darwin in conjunction with the 200th anniversary of his birth.

...

System Message: ERROR/3 (<stdin>, line 97)

Unknown directive type "highlight".

.. highlight:: none

An XPath expression to select the description could be:

//div[@id='description']

System Message: ERROR/3 (<stdin>, line 103)

Unknown directive type "highlight".

.. highlight:: html

Finally, the file size is contained in the second <p> tag inside the <div> tag with id=specifications:

<div id="specifications">

<p>
<strong>Category:</strong>
<a href="/cat/4">Movies</a> &gt; <a href="/sub/35">Documentary</a>
</p>

<p>
<strong>Total size:</strong>
150.62&nbsp;megabyte</p>

System Message: ERROR/3 (<stdin>, line 120)

Unknown directive type "highlight".

.. highlight:: none

An XPath expression to select the file size could be:

//div[@id='specifications']/p[2]/text()[2]

System Message: ERROR/3 (<stdin>, line 126)

Unknown directive type "highlight".

.. highlight:: python

For more information about XPath see the XPath reference.

Finally, here's the spider code:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

class MininovaSpider(CrawlSpider):

    name = 'mininova'
    allowed_domains = ['mininova.org']
    start_urls = ['http://www.mininova.org/today']
    rules = [Rule(LinkExtractor(allow=['/tor/\d+']), 'parse_torrent')]

    def parse_torrent(self, response):
        torrent = TorrentItem()
        torrent['url'] = response.url
        torrent['name'] = response.xpath("//h1/text()").extract()
        torrent['description'] = response.xpath("//div[@id='description']").extract()
        torrent['size'] = response.xpath("//div[@id='specifications']/p[2]/text()[2]").extract()
        return torrent

The TorrentItem class is :ref:`defined above <intro-overview-item>`.

System Message: ERROR/3 (<stdin>, line 150); backlink

Unknown interpreted text role "ref".

Run the spider to extract the data

Finally, we'll run the spider to crawl the site and output the file scraped_data.json with the scraped data in JSON format:

scrapy crawl mininova -o scraped_data.json

This uses :ref:`feed exports <topics-feed-exports>` to generate the JSON file. You can easily change the export format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example).

System Message: ERROR/3 (<stdin>, line 160); backlink

Unknown interpreted text role "ref".

You can also write an :ref:`item pipeline <topics-item-pipeline>` to store the items in a database very easily.

System Message: ERROR/3 (<stdin>, line 164); backlink

Unknown interpreted text role "ref".

Review scraped data

If you check the scraped_data.json file after the process finishes, you'll see the scraped items there:

[{"url": "http://www.mininova.org/tor/2676093", "name": ["Darwin - The Evolution Of An Exhibition"], "description": ["Short documentary made for Plymouth ..."], "size": ["150.62 megabyte"]},
# ... other items ...
]

You'll notice that all field values (except for the url which was assigned directly) are actually lists. This is because the :ref:`selectors <topics-selectors>` return lists. You may want to store single values, or perform some additional parsing/cleansing to the values. That's what :ref:`Item Loaders <topics-loaders>` are for.

System Message: ERROR/3 (<stdin>, line 177); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 177); backlink

Unknown interpreted text role "ref".

What else?

You've seen how to extract and store items from a website using Scrapy, but this is just the surface. Scrapy provides a lot of powerful features for making scraping easy and efficient, such as:

What's next?

The next obvious steps are for you to download Scrapy, read :ref:`the tutorial <intro-tutorial>` and join the community. Thanks for your interest!

System Message: ERROR/3 (<stdin>, line 256); backlink

Unknown interpreted text role "ref".
</html>