From e3d67d74f73045f82ec9aae11439020c87a4b371 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Mon, 6 Sep 2010 10:04:00 -0300 Subject: [PATCH] docs/intro/overview.rst: add example of scraped data and introduce loaders --- docs/intro/overview.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 47cae7904..7b4f490a0 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -170,6 +170,22 @@ storage backend (FTP or `Amazon S3`_, for example). You can also write an :ref:`item pipeline ` to store the items in a database very easily. +Review scraped data +=================== + +If you check the ``scraped_data.json`` file after the process finishes, you'll +see the scraped items there:: + + [{"url": "http://www.mininova.org/tor/2657665", "name": ["Home[2009][Eng]XviD-ovd"], "description": ["HOME - a documentary film by ..."], "size": ["699.69 megabyte"]}, + # ... other items ... + ] + +You'll notice that all field values (except for the ``url`` which was assigned +directly) are actually lists. This is because the :ref:`selectors +` return lists. You may want to store single values, or +perform some additional parsing/cleansing to the values. That's what +:ref:`Item Loaders ` are for. + What else? ========== @@ -181,7 +197,7 @@ scraping easy and efficient, such as: from HTML and XML sources * Built-in support for cleaning and sanitizing the scraped data using a - collection of reusable filters (called :ref:`loaders `) + collection of reusable filters (called :ref:`Item Loaders `) shared between all the spiders. * Built-in support for :ref:`generating feed exports ` in