mirror of https://github.com/scrapy/scrapy.git
some improvements to overview page
This commit is contained in:
Elias Dorneles
parent
e3e60da6c2
commit
32423d4a33
|
|
@ -13,172 +13,83 @@ precisely, `web scraping`_), it can also be used to extract data using APIs
|
|||
(such as `Amazon Associates Web Services`_) or as a general purpose web
|
||||
crawler.
|
||||
|
||||
The purpose of this document is to introduce you to the concepts behind Scrapy
|
||||
so you can get an idea of how it works and decide if Scrapy is what you need.
|
||||
|
||||
When you're ready to start a project, you can :ref:`start with the tutorial
|
||||
<intro-tutorial>`.
|
||||
Walk-through of an example spider
|
||||
=================================
|
||||
|
||||
Pick a website
|
||||
==============
|
||||
In order to show you what Scrapy brings to the table, we'll walk you
|
||||
through an example of a Scrapy Spider using the simplest way to run a spider.
|
||||
|
||||
So you need to extract some information from a website, but the website doesn't
|
||||
provide any API or mechanism to access that info programmatically. Scrapy can
|
||||
help you extract that information.
|
||||
Once you're ready to dive in more, you can :ref:`follow the tutorial
|
||||
and build a full-blown Scrapy project <intro-tutorial>`.
|
||||
|
||||
Let's say we want to extract the URL, name, description and size of all torrent
|
||||
files added today in the `Mininova`_ site.
|
||||
|
||||
The list of all torrents added today can be found on this page:
|
||||
|
||||
http://www.mininova.org/today
|
||||
|
||||
.. _intro-overview-item:
|
||||
|
||||
Define the data you want to scrape
|
||||
==================================
|
||||
|
||||
The first thing is to define the data we want to scrape. In Scrapy, this is
|
||||
done through :ref:`Scrapy Items <topics-items>` (Torrent files, in this case).
|
||||
|
||||
This would be our Item::
|
||||
So, here's the code for a spider that follows the links to the top
|
||||
voted questions on StackOverflow and scrapes some data from each page::
|
||||
|
||||
import scrapy
|
||||
|
||||
class TorrentItem(scrapy.Item):
|
||||
url = scrapy.Field()
|
||||
name = scrapy.Field()
|
||||
description = scrapy.Field()
|
||||
size = scrapy.Field()
|
||||
|
||||
Write a Spider to extract the data
|
||||
==================================
|
||||
class StackOverflowSpider(scrapy.Spider):
|
||||
name = 'stackoverflow'
|
||||
start_urls = ['http://stackoverflow.com/questions?sort=votes']
|
||||
|
||||
The next thing is to write a Spider which defines the start URL
|
||||
(http://www.mininova.org/today), the rules for following links and the rules
|
||||
for extracting the data from pages.
|
||||
def parse(self, response):
|
||||
for href in response.css('.question-summary h3 a::attr(href)'):
|
||||
full_url = response.urljoin(href.extract())
|
||||
yield scrapy.Request(full_url, callback=self.parse_question)
|
||||
|
||||
If we take a look at that page content we'll see that all torrent URLs are like
|
||||
``http://www.mininova.org/tor/NUMBER`` where ``NUMBER`` is an integer. We'll use
|
||||
that to construct the regular expression for the links to follow: ``/tor/\d+``.
|
||||
|
||||
We'll use `XPath`_ for selecting the data to extract from the web page HTML
|
||||
source. Let's take one of those torrent pages:
|
||||
|
||||
http://www.mininova.org/tor/2676093
|
||||
|
||||
And look at the page HTML source to construct the XPath to select the data we
|
||||
want which is: torrent name, description and size.
|
||||
|
||||
.. highlight:: html
|
||||
|
||||
By looking at the page HTML source we can see that the file name is contained
|
||||
inside a ``<h1>`` tag::
|
||||
|
||||
<h1>Darwin - The Evolution Of An Exhibition</h1>
|
||||
|
||||
.. highlight:: none
|
||||
|
||||
An XPath expression to extract the name could be::
|
||||
|
||||
//h1/text()
|
||||
|
||||
.. highlight:: html
|
||||
|
||||
And the description is contained inside a ``<div>`` tag with ``id="description"``::
|
||||
|
||||
<h2>Description:</h2>
|
||||
|
||||
<div id="description">
|
||||
Short documentary made for Plymouth City Museum and Art Gallery regarding the setup of an exhibit about Charles Darwin in conjunction with the 200th anniversary of his birth.
|
||||
|
||||
...
|
||||
|
||||
.. highlight:: none
|
||||
|
||||
An XPath expression to select the description could be::
|
||||
|
||||
//div[@id='description']
|
||||
|
||||
.. highlight:: html
|
||||
|
||||
Finally, the file size is contained in the second ``<p>`` tag inside the ``<div>``
|
||||
tag with ``id=specifications``::
|
||||
|
||||
<div id="specifications">
|
||||
|
||||
<p>
|
||||
<strong>Category:</strong>
|
||||
<a href="/cat/4">Movies</a> > <a href="/sub/35">Documentary</a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<strong>Total size:</strong>
|
||||
150.62 megabyte</p>
|
||||
def parse_question(self, response):
|
||||
title = response.css('h1 a::text').extract_first()
|
||||
votes = response.css('.question .vote-count-post::text').extract_first()
|
||||
tags = response.css('.question .post-tag::text').extract()
|
||||
body = response.css('.question .post-text').extract_first()
|
||||
yield {
|
||||
'title': title,
|
||||
'votes': votes,
|
||||
'body': body,
|
||||
'tags': tags,
|
||||
'link': response.url,
|
||||
}
|
||||
|
||||
|
||||
.. highlight:: none
|
||||
Put this in a file, name it to something like ``stackoverflow_spider.py``
|
||||
and run the spider using the :command:`runspider` command::
|
||||
|
||||
An XPath expression to select the file size could be::
|
||||
scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json
|
||||
|
||||
//div[@id='specifications']/p[2]/text()[2]
|
||||
|
||||
.. highlight:: python
|
||||
When this finishes you will have in the ``top-stackoverflow-questions.json`` file
|
||||
a list of the most upvoted questions in StackOverflow in JSON format, containing the
|
||||
title, link, number of upvotes, a list of the tags and the question content in HTML.
|
||||
|
||||
For more information about XPath see the `XPath reference`_.
|
||||
|
||||
Finally, here's the spider code::
|
||||
What just happened?
|
||||
-------------------
|
||||
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.contrib.linkextractors import LinkExtractor
|
||||
When you ran the command ``scrapy runspider somefile.py``, Scrapy looked
|
||||
for a Spider definition inside it and ran it through its crawler engine.
|
||||
|
||||
class MininovaSpider(CrawlSpider):
|
||||
The crawl started by making requests to the URLs defined in the ``start_urls``
|
||||
attribute (in this case, only the URL for StackOverflow top questions page),
|
||||
and then called the default callback method ``parse`` passing the response
|
||||
object as an argument.
|
||||
|
||||
name = 'mininova'
|
||||
allowed_domains = ['mininova.org']
|
||||
start_urls = ['http://www.mininova.org/today']
|
||||
rules = [Rule(LinkExtractor(allow=['/tor/\d+']), 'parse_torrent')]
|
||||
In the ``parse`` callback, we scrape the links to the questions and
|
||||
yield a few more requests to be processed, registering for them
|
||||
the method ``parse_question`` as the callback to be called when the
|
||||
requests are complete.
|
||||
|
||||
def parse_torrent(self, response):
|
||||
torrent = TorrentItem()
|
||||
torrent['url'] = response.url
|
||||
torrent['name'] = response.xpath("//h1/text()").extract()
|
||||
torrent['description'] = response.xpath("//div[@id='description']").extract()
|
||||
torrent['size'] = response.xpath("//div[@id='info-left']/p[2]/text()[2]").extract()
|
||||
return torrent
|
||||
Finally, the ``parse_question`` callback scrapes the question data
|
||||
for each page yielding a dict, which Scrapy then collects and
|
||||
writes to a JSON file as requested in the command line.
|
||||
|
||||
The ``TorrentItem`` class is :ref:`defined above <intro-overview-item>`.
|
||||
.. note::
|
||||
|
||||
Run the spider to extract the data
|
||||
==================================
|
||||
This is using :ref:`feed exports <topics-feed-exports>` to generate the
|
||||
JSON file, you can easily change the export format (XML or CSV, for example) or the
|
||||
storage backend (FTP or `Amazon S3`_, for example). You can also write an
|
||||
:ref:`item pipeline <topics-item-pipeline>` to store the items in a database.
|
||||
|
||||
Finally, we'll run the spider to crawl the site and output the file
|
||||
``scraped_data.json`` with the scraped data in JSON format::
|
||||
|
||||
scrapy crawl mininova -o scraped_data.json
|
||||
|
||||
This uses :ref:`feed exports <topics-feed-exports>` to generate the JSON file.
|
||||
You can easily change the export format (XML or CSV, for example) or the
|
||||
storage backend (FTP or `Amazon S3`_, for example).
|
||||
|
||||
You can also write an :ref:`item pipeline <topics-item-pipeline>` to store the
|
||||
items in a database very easily.
|
||||
|
||||
Review scraped data
|
||||
===================
|
||||
|
||||
If you check the ``scraped_data.json`` file after the process finishes, you'll
|
||||
see the scraped items there::
|
||||
|
||||
[{"url": "http://www.mininova.org/tor/2676093", "name": ["Darwin - The Evolution Of An Exhibition"], "description": ["Short documentary made for Plymouth ..."], "size": ["150.62 megabyte"]},
|
||||
# ... other items ...
|
||||
]
|
||||
|
||||
You'll notice that all field values (except for the ``url`` which was assigned
|
||||
directly) are actually lists. This is because the :ref:`selectors
|
||||
<topics-selectors>` return lists. You may want to store single values, or
|
||||
perform some additional parsing/cleansing to the values. That's what
|
||||
:ref:`Item Loaders <topics-loaders>` are for.
|
||||
|
||||
.. _topics-whatelse:
|
||||
|
||||
|
|
@ -189,68 +100,37 @@ You've seen how to extract and store items from a website using Scrapy, but
|
|||
this is just the surface. Scrapy provides a lot of powerful features for making
|
||||
scraping easy and efficient, such as:
|
||||
|
||||
* Built-in support for :ref:`selecting and extracting <topics-selectors>` data
|
||||
from HTML and XML sources
|
||||
|
||||
* Built-in support for cleaning and sanitizing the scraped data using a
|
||||
collection of reusable filters (called :ref:`Item Loaders <topics-loaders>`)
|
||||
shared between all the spiders.
|
||||
* An :ref:`interactive shell console <topics-shell>` (IPython aware) for trying
|
||||
out the CSS and XPath expressions to scrape data, very useful when writing or
|
||||
debugging your spiders.
|
||||
|
||||
* Built-in support for :ref:`generating feed exports <topics-feed-exports>` in
|
||||
multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP,
|
||||
S3, local filesystem)
|
||||
|
||||
* A media pipeline for :ref:`automatically downloading images <topics-images>`
|
||||
(or any other media) associated with the scraped items
|
||||
|
||||
* Support for :ref:`extending Scrapy <extending-scrapy>` by plugging
|
||||
your own functionality using :ref:`signals <topics-signals>` and a
|
||||
well-defined API (middlewares, :ref:`extensions <topics-extensions>`, and
|
||||
:ref:`pipelines <topics-item-pipeline>`).
|
||||
|
||||
* Wide range of built-in middlewares and extensions for:
|
||||
|
||||
* cookies and session handling
|
||||
* HTTP compression
|
||||
* HTTP authentication
|
||||
* HTTP cache
|
||||
* user-agent spoofing
|
||||
* robots.txt
|
||||
* crawl depth restriction
|
||||
* and more
|
||||
|
||||
* Robust encoding support and auto-detection, for dealing with foreign,
|
||||
non-standard and broken encoding declarations.
|
||||
|
||||
* Support for creating spiders based on pre-defined templates, to speed up
|
||||
spider creation and make their code more consistent on large projects. See
|
||||
:command:`genspider` command for more details.
|
||||
|
||||
* Extensible :ref:`stats collection <topics-stats>` for multiple spider
|
||||
metrics, useful for monitoring the performance of your spiders and detecting
|
||||
when they get broken
|
||||
|
||||
* An :ref:`Interactive shell console <topics-shell>` for trying XPaths, very
|
||||
useful for writing and debugging your spiders
|
||||
|
||||
* A :ref:`System service <topics-scrapyd>` designed to ease the deployment and
|
||||
run of your spiders in production.
|
||||
* Strong :ref:`extensibility support <extending-scrapy>` and lots of built-in
|
||||
extensions and middlewares to handle things like cookies, crawl throttling,
|
||||
HTTP caching, HTTP compression, user-agent spoofing, robots.txt,
|
||||
stats collection and many more.
|
||||
|
||||
* A :ref:`Telnet console <topics-telnetconsole>` for hooking into a Python
|
||||
console running inside your Scrapy process, to introspect and debug your
|
||||
crawler
|
||||
|
||||
* :ref:`Logging <topics-logging>` facility that you can hook on to for catching
|
||||
errors during the scraping process.
|
||||
* A caching DNS resolver
|
||||
|
||||
* Support for crawling based on URLs discovered through `Sitemaps`_
|
||||
|
||||
* A caching DNS resolver
|
||||
* A media pipeline for :ref:`automatically downloading images <topics-images>`
|
||||
(or any other media) associated with the scraped items
|
||||
|
||||
What's next?
|
||||
============
|
||||
|
||||
The next obvious steps are for you to `download Scrapy`_, read :ref:`the
|
||||
The next obvious steps for you are to `download Scrapy`_, read :ref:`the
|
||||
tutorial <intro-tutorial>` and join `the community`_. Thanks for your
|
||||
interest!
|
||||
|
||||
|
|
@ -258,9 +138,6 @@ interest!
|
|||
.. _the community: http://scrapy.org/community/
|
||||
.. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping
|
||||
.. _web scraping: http://en.wikipedia.org/wiki/Web_scraping
|
||||
.. _Amazon Associates Web Services: https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html
|
||||
.. _Mininova: http://www.mininova.org
|
||||
.. _XPath: http://www.w3.org/TR/xpath
|
||||
.. _XPath reference: http://www.w3.org/TR/xpath
|
||||
.. _Amazon Associates Web Services: http://aws.amazon.com/associates/
|
||||
.. _Amazon S3: http://aws.amazon.com/s3/
|
||||
.. _Sitemaps: http://www.sitemaps.org
|
||||
|
|
|
|||
Loading…
Reference in New Issue