RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated 'Scrapy at a glance' document replacing item pipeline example by a simpler usage of feed exports

This commit is contained in:
Pablo Hoffman 2010-09-05 23:38:37 -03:00
parent 5ffc7650bd
commit 00d55fbbd1
1 changed files with 69 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

112
docs/intro/overview.rst
View File

@ -13,10 +13,6 @@ precisely, `web scraping`_), it can also be used to extract data using APIs
(such as `Amazon Associates Web Services`_) or as a general purpose web
crawler.
.. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping
.. _web scraping: http://en.wikipedia.org/wiki/Web_scraping
.. _Amazon Associates Web Services: http://aws.amazon.com/associates/
The purpose of this document is to introduce you to the concepts behind Scrapy
so you can get an idea of how it works and decide if Scrapy is what you need.
@ -27,21 +23,38 @@ Pick a website
==============
So you need to extract some information from a website, but the website doesn't
provide any API or mechanism to access that info from a computer program.
Scrapy can help you extract that information. Let's say we want to extract
information about all torrent files added today in the `mininova`_ torrent
site.
provide any API or mechanism to access that info programmatically. Scrapy can
help you extract that information.
.. _mininova: http://www.mininova.org
Let's say we want to extract the URL, name, description and size of all torrent
files added today in the `Mininova`_ site.
The list of all torrents added today can be found in this page:
The list of all torrents added today can be found on this page:
http://www.mininova.org/today
Write a Spider to extract the Items
===================================
.. _intro-overview-item:
Now we'll write a Spider which defines the start URL
Define the data you want to scrape
==================================
The first thing is to define the data we want to scrape. In Scrapy, this is
done through :ref:`Scrapy Items <topics-items>` (Torrent files, in this case).
This would be our Item::
from scrapy.item import Item
class Torrent(Item):
url = Field()
name = Field()
description = Field()
size = Field()
Write a Spider to extract the data
==================================
The next thing is to write a Spider which defines the start URL
(http://www.mininova.org/today), the rules for following links and the rules
for extracting the data from pages.
@ -49,13 +62,11 @@ If we take a look at that page content we'll see that all torrent URLs are like
http://www.mininova.org/tor/NUMBER where ``NUMBER`` is an integer. We'll use
that to construct the regular expression for the links to follow: ``/tor/\d+``.
To extracting data, we'll use `XPath`_ to select the part of the document where
the data is to be extracted from. Let's take one of those torrent pages:
We'll use `XPath`_ for selecting the data to extract from the web page HTML
source. Let's take one of those torrent pages:
http://www.mininova.org/tor/2657665
.. _XPath: http://www.w3.org/TR/xpath
And look at the page HTML source to construct the XPath to select the data we
want which is: torrent name, description and size.
@ -122,8 +133,6 @@ An XPath expression to select the description could be::
For more information about XPath see the `XPath reference`_.
.. _XPath reference: http://www.w3.org/TR/xpath
Finally, here's the spider code::
class MininovaSpider(CrawlSpider):
@ -143,26 +152,23 @@ Finally, here's the spider code::
torrent['size'] = x.select("//div[@id='info-left']/p[2]/text()[2]").extract()
return torrent
For brevity's sake, we intentionally left out the import statements. The
Torrent item is :ref:`defined above <intro-overview-item>`.
For brevity's sake, we intentionally left out the import statements and the
Torrent class definition (which is included some paragraphs above).
Run the spider to extract the data
==================================
Write a pipeline to store the items extracted
=============================================
Finally, we'll run the spider to crawl the site an output file
``scraped_data.json`` with the scraped data in JSON format::
Now let's write an :ref:`topics-item-pipeline` that serializes and stores the
extracted item into a file using `pickle`_::
scrapy crawl mininova.org --set FEED_URI=scraped_data.json --set FEED_FORMAT=json
import pickle
This uses :ref:`feed exports <topics-feed-exports>` to generate the JSON file.
You can easily change the export format (XML or CSV, for example) or the
storage backend (FTP or `Amazon S3`_, for example).
class StoreItemPipeline(object):
def process_item(self, item, spider):
torrent_id = item['url'].split('/')[-1]
f = open("torrent-%s.pickle" % torrent_id, "w")
pickle.dump(item, f)
f.close()
.. _pickle: http://docs.python.org/library/pickle.html
You can also write an :ref:`item pipeline <topics-item-pipeline>` to store the
items in a database very easily.
What else?
==========
@ -174,32 +180,45 @@ scraping easy and efficient, such as:
* Built-in support for :ref:`selecting and extracting <topics-selectors>` data
from HTML and XML sources
* Built-in support for cleaning and sanitizing the scraped data using a
collection of reusable filters (called :ref:`loaders <topics-loaders>`)
shared between all the spiders.
* Built-in support for :ref:`generating feed exports <topics-feed-exports>` in
multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP,
S3, filesystem)
S3, local filesystem)
* A media pipeline for :ref:`automatically downloading images <topics-images>`
(or any other media) associated with the scraped items
* Support for :ref:`extending Scrapy <extending-scrapy>` by plugging
your own functionality using middlewares, extensions, and pipelines
your own functionality using :ref:`signals <topics-signals>` and a
well-defined API (middlewares, :ref:`extensions <topics-extensions>`, and
:ref:`pipelines <topics-item-pipeline>`).
* Wide range of built-in middlewares and extensions for handling of
compression, cache, cookies, authentication, user-agent spoofing, robots.txt
handling, statistics, crawl depth restriction, etc
* An :ref:`Interactive scraping shell console <topics-shell>`, very useful for
writing and debugging your spiders
* Extensible :ref:`stats collection <topics-stats>` for multiple spider
metrics, useful for monitoring the performance of your spiders and detecting
when they get broken
* A builtin :ref:`Web service <topics-webservice>` for monitoring and
* An :ref:`Interactive shell console <topics-shell>` for trying XPaths, very
useful for writing and debugging your spiders
* A :ref:`System service <topics-scrapyd>` designed to ease the deployment and
run of your spiders in production.
* A built-in :ref:`Web service <topics-webservice>` for monitoring and
controlling your bot
* A :ref:`Telnet console <topics-telnetconsole>` for full unrestricted access
to a Python console inside your Scrapy process, to introspect and debug your
* A :ref:`Telnet console <topics-telnetconsole>` for hooking into a Python
console running inside your Scrapy process, to introspect and debug your
crawler
* Built-in facilities for :ref:`logging <topics-logging>`, :ref:`collecting
stats <topics-stats>`, and :ref:`sending email notifications <topics-email>`
* Built-in :ref:`logging <topics-logging>` facility that you can hook to for
catching errors during the scraping process.
What's next?
============
@ -210,3 +229,10 @@ interest!
.. _download Scrapy: http://scrapy.org/download/
.. _the community: http://scrapy.org/community/
.. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping
.. _web scraping: http://en.wikipedia.org/wiki/Web_scraping
.. _Amazon Associates Web Services: http://aws.amazon.com/associates/
.. _Mininova: http://www.mininova.org
.. _XPath: http://www.w3.org/TR/xpath
.. _XPath reference: http://www.w3.org/TR/xpath
.. _Amazon S3: http://aws.amazon.com/s3/