2.6 KiB

Raw Blame History

Finishing the job

Well, we've got our project, our spider, and our scraped items. What to do next?

It actually depends on what you want to do with the scraped data.

In this case, we'll imagine that we want to save this data for storing it in a db later, or just to keep it there.

To make it simple, we'll export the scraped items to a CSV file by making use of a useful function that Scrapy brings: items_to_csv. This simple function takes a file descriptor/filename, and a list of items, and writes their attributes to that file, in CSV format.

Let's see how would our spider end up looking like after applying this change:

# -*- coding: utf8 -*-
from scrapy.xpath import HtmlXPathSelector
from scrapy.item import ScrapedItem
from scrapy.contrib import adaptors
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.link.extractors import RegexLinkExtractor
from scrapy.utils.misc import items_to_csv

class GoogleDirectorySpider(CrawlSpider):
    domain_name = 'google.com'
    start_urls = ['http://www.google.com/dirhp']

    rules = (
        Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$', ), ),
            'parse_category',
            follow=True,
        ),
    )
    csv_file = open('scraped_items.csv', 'w')

    def parse_category(self, response):
        items = [] # The item (links to websites) list we're going to return
        hxs = HtmlXPathSelector(response) # The selector we're going to use in order to extract data from the page
        links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')

        for link in links:
            item = ScrapedItem()
            adaptor_pipe = [adaptors.extract, adaptors.Delist(''), adaptors.strip]
            item.set_adaptors({
                'name': adaptor_pipe,
                'url': adaptor_pipe,
                'description': adaptor_pipe,
            })

            item.attribute('name', link.x('a/text()'))
            item.attribute('url', link.x('a/@href'))
            item.attribute('description', link.x('font[2]/text()'))
            items.append(item)

        items_to_csv(self.csv_file, items)
        return items

SPIDER = GoogleDirectorySpider()

With this code, our spider will crawl over Google's directory, and save each link's name, description, and url to a file called 'scraped_items.csv'. Cool, huh?

This is the end of the tutorial. If you'd like to know more about Scrapy and its use, please read the rest of the documentation.

</html>

2.6 KiB Raw Blame History

Finishing the job

2.6 KiB

Raw Blame History