From 00d55fbbd10812d59ac993253e453c9378a104db Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Sun, 5 Sep 2010 23:38:37 -0300 Subject: [PATCH] Updated 'Scrapy at a glance' document replacing item pipeline example by a simpler usage of feed exports --- docs/intro/overview.rst | 112 +++++++++++++++++++++++++--------------- 1 file changed, 69 insertions(+), 43 deletions(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 08c1de0b8..47cae7904 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -13,10 +13,6 @@ precisely, `web scraping`_), it can also be used to extract data using APIs (such as `Amazon Associates Web Services`_) or as a general purpose web crawler. -.. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping -.. _web scraping: http://en.wikipedia.org/wiki/Web_scraping -.. _Amazon Associates Web Services: http://aws.amazon.com/associates/ - The purpose of this document is to introduce you to the concepts behind Scrapy so you can get an idea of how it works and decide if Scrapy is what you need. @@ -27,21 +23,38 @@ Pick a website ============== So you need to extract some information from a website, but the website doesn't -provide any API or mechanism to access that info from a computer program. -Scrapy can help you extract that information. Let's say we want to extract -information about all torrent files added today in the `mininova`_ torrent -site. +provide any API or mechanism to access that info programmatically. Scrapy can +help you extract that information. -.. _mininova: http://www.mininova.org +Let's say we want to extract the URL, name, description and size of all torrent +files added today in the `Mininova`_ site. -The list of all torrents added today can be found in this page: +The list of all torrents added today can be found on this page: http://www.mininova.org/today -Write a Spider to extract the Items -=================================== +.. _intro-overview-item: -Now we'll write a Spider which defines the start URL +Define the data you want to scrape +================================== + +The first thing is to define the data we want to scrape. In Scrapy, this is +done through :ref:`Scrapy Items ` (Torrent files, in this case). + +This would be our Item:: + + from scrapy.item import Item + + class Torrent(Item): + url = Field() + name = Field() + description = Field() + size = Field() + +Write a Spider to extract the data +================================== + +The next thing is to write a Spider which defines the start URL (http://www.mininova.org/today), the rules for following links and the rules for extracting the data from pages. @@ -49,13 +62,11 @@ If we take a look at that page content we'll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where ``NUMBER`` is an integer. We'll use that to construct the regular expression for the links to follow: ``/tor/\d+``. -To extracting data, we'll use `XPath`_ to select the part of the document where -the data is to be extracted from. Let's take one of those torrent pages: +We'll use `XPath`_ for selecting the data to extract from the web page HTML +source. Let's take one of those torrent pages: http://www.mininova.org/tor/2657665 -.. _XPath: http://www.w3.org/TR/xpath - And look at the page HTML source to construct the XPath to select the data we want which is: torrent name, description and size. @@ -122,8 +133,6 @@ An XPath expression to select the description could be:: For more information about XPath see the `XPath reference`_. -.. _XPath reference: http://www.w3.org/TR/xpath - Finally, here's the spider code:: class MininovaSpider(CrawlSpider): @@ -143,26 +152,23 @@ Finally, here's the spider code:: torrent['size'] = x.select("//div[@id='info-left']/p[2]/text()[2]").extract() return torrent +For brevity's sake, we intentionally left out the import statements. The +Torrent item is :ref:`defined above `. -For brevity's sake, we intentionally left out the import statements and the -Torrent class definition (which is included some paragraphs above). +Run the spider to extract the data +================================== -Write a pipeline to store the items extracted -============================================= +Finally, we'll run the spider to crawl the site an output file +``scraped_data.json`` with the scraped data in JSON format:: -Now let's write an :ref:`topics-item-pipeline` that serializes and stores the -extracted item into a file using `pickle`_:: + scrapy crawl mininova.org --set FEED_URI=scraped_data.json --set FEED_FORMAT=json - import pickle +This uses :ref:`feed exports ` to generate the JSON file. +You can easily change the export format (XML or CSV, for example) or the +storage backend (FTP or `Amazon S3`_, for example). - class StoreItemPipeline(object): - def process_item(self, item, spider): - torrent_id = item['url'].split('/')[-1] - f = open("torrent-%s.pickle" % torrent_id, "w") - pickle.dump(item, f) - f.close() - -.. _pickle: http://docs.python.org/library/pickle.html +You can also write an :ref:`item pipeline ` to store the +items in a database very easily. What else? ========== @@ -174,32 +180,45 @@ scraping easy and efficient, such as: * Built-in support for :ref:`selecting and extracting ` data from HTML and XML sources +* Built-in support for cleaning and sanitizing the scraped data using a + collection of reusable filters (called :ref:`loaders `) + shared between all the spiders. + * Built-in support for :ref:`generating feed exports ` in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, - S3, filesystem) + S3, local filesystem) * A media pipeline for :ref:`automatically downloading images ` (or any other media) associated with the scraped items * Support for :ref:`extending Scrapy ` by plugging - your own functionality using middlewares, extensions, and pipelines + your own functionality using :ref:`signals ` and a + well-defined API (middlewares, :ref:`extensions `, and + :ref:`pipelines `). * Wide range of built-in middlewares and extensions for handling of compression, cache, cookies, authentication, user-agent spoofing, robots.txt handling, statistics, crawl depth restriction, etc -* An :ref:`Interactive scraping shell console `, very useful for - writing and debugging your spiders +* Extensible :ref:`stats collection ` for multiple spider + metrics, useful for monitoring the performance of your spiders and detecting + when they get broken -* A builtin :ref:`Web service ` for monitoring and +* An :ref:`Interactive shell console ` for trying XPaths, very + useful for writing and debugging your spiders + +* A :ref:`System service ` designed to ease the deployment and + run of your spiders in production. + +* A built-in :ref:`Web service ` for monitoring and controlling your bot -* A :ref:`Telnet console ` for full unrestricted access - to a Python console inside your Scrapy process, to introspect and debug your +* A :ref:`Telnet console ` for hooking into a Python + console running inside your Scrapy process, to introspect and debug your crawler -* Built-in facilities for :ref:`logging `, :ref:`collecting - stats `, and :ref:`sending email notifications ` +* Built-in :ref:`logging ` facility that you can hook to for + catching errors during the scraping process. What's next? ============ @@ -210,3 +229,10 @@ interest! .. _download Scrapy: http://scrapy.org/download/ .. _the community: http://scrapy.org/community/ +.. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping +.. _web scraping: http://en.wikipedia.org/wiki/Web_scraping +.. _Amazon Associates Web Services: http://aws.amazon.com/associates/ +.. _Mininova: http://www.mininova.org +.. _XPath: http://www.w3.org/TR/xpath +.. _XPath reference: http://www.w3.org/TR/xpath +.. _Amazon S3: http://aws.amazon.com/s3/