DOC mention dicts in documentation; explain better what are Items for

docs/topics/architecture.rst

@@ -102,10 +102,10 @@ this:
 6. The Engine receives the Response from the Downloader and sends it to the
    Spider for processing, passing through the Spider Middleware (input direction).
 
-7. The Spider processes the Response and returns scraped Items and new Requests
+7. The Spider processes the Response and returns scraped items and new Requests
    (to follow) to the Engine.
 
-8. The Engine sends scraped Items (returned by the Spider) to the Item Pipeline
+8. The Engine sends scraped items (returned by the Spider) to the Item Pipeline
    and Requests (returned by spider) to the Scheduler
 
 9. The process repeats (from step 2) until there are no more requests from the

docs/topics/exporters.rst

@@ -7,7 +7,7 @@ Item Exporters
 .. module:: scrapy.contrib.exporter
    :synopsis: Item Exporters
 
-Once you have scraped your Items, you often want to persist or export those
+Once you have scraped your items, you often want to persist or export those
 items, to use the data in some other application. That is, after all, the whole
 purpose of the scraping process.
 
@@ -90,9 +90,9 @@ described next.
 1. Declaring a serializer in the field
 --------------------------------------
 
-You can declare a serializer in the :ref:`field metadata
-<topics-items-fields>`. The serializer must be a callable which receives a
-value and returns its serialized form.
+If you use :class:`~.Item` you can declare a serializer in the 
+:ref:`field metadata <topics-items-fields>`. The serializer must be 
+a callable which receives a value and returns its serialized form.
 
 Example::
 
@@ -167,8 +167,9 @@ BaseItemExporter
       value unchanged except for ``unicode`` values which are encoded to
       ``str`` using the encoding declared in the :attr:`encoding` attribute.
 
-      :param field: the field being serialized
-      :type field: :class:`~scrapy.item.Field` object
+      :param field: the field being serialized. If a raw dict is being 
+          exported (not :class:`~.Item`) *field* value is an empty dict.
+      :type field: :class:`~scrapy.item.Field` object or an empty dict 
 
       :param name: the name of the field being serialized
       :type name: str

docs/topics/images.rst

@@ -63,9 +63,14 @@ this:
 Usage example
 =============
 
-In order to use the image pipeline you just need to :ref:`enable it
-<topics-images-enabling>` and define an item with the ``image_urls`` and
-``images`` fields::
+In order to use the image pipeline first  
+:ref:`enable it <topics-images-enabling>`.
+
+Then, if a spider returns a dict with 'image_urls' key, 
+the pipeline will put the results under 'images' key.
+
+If you prefer to use :class:`~.Item` then define a custom 
+item with the ``image_urls`` and ``images`` fields::
 
     import scrapy
 
@@ -74,7 +79,7 @@ In order to use the image pipeline you just need to :ref:`enable it
         # ... other item fields ...
         image_urls = scrapy.Field()
         images = scrapy.Field()
-
+        
 If you need something more complex and want to override the custom images
 pipeline behaviour, see :ref:`topics-images-override`.
 

docs/topics/item-pipeline.rst

@@ -8,8 +8,8 @@ After an item has been scraped by a spider, it is sent to the Item Pipeline
 which process it through several components that are executed sequentially.
 
 Each item pipeline component (sometimes referred as just "Item Pipeline") is a
-Python class that implements a simple method. They receive an Item and perform
-an action over it, also deciding if the Item should continue through the
+Python class that implements a simple method. They receive an item and perform
+an action over it, also deciding if the item should continue through the
 pipeline or be dropped and no longer processed.
 
 Typical use for item pipelines are:
@@ -28,12 +28,12 @@ Each item pipeline component is a Python class that must implement the following
 .. method:: process_item(self, item, spider)
 
    This method is called for every item pipeline component and must either return
-   a :class:`~scrapy.item.Item` (or any descendant class) object or raise a
-   :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer
+   a dict with data, :class:`~scrapy.item.Item` (or any descendant class) object 
+   or raise a :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer
    processed by further pipeline components.
 
    :param item: the item scraped
-   :type item: :class:`~scrapy.item.Item` object
+   :type item: :class:`~scrapy.item.Item` object or a dict
 
    :param spider: the spider which scraped the item
    :type spider: :class:`~scrapy.spider.Spider` object
@@ -135,6 +135,8 @@ method and how to clean up the resources properly.
     import pymongo
 
     class MongoPipeline(object):
+    
+        collection_name = 'scrapy_items'
 
         def __init__(self, mongo_uri, mongo_db):
             self.mongo_uri = mongo_uri
@@ -155,8 +157,7 @@ method and how to clean up the resources properly.
             self.client.close()
 
         def process_item(self, item, spider):
-            collection_name = item.__class__.__name__
-            self.db[collection_name].insert(dict(item))
+            self.db[self.collection_name].insert(dict(item))
             return item
 
 .. _MongoDB: http://www.mongodb.org/

docs/topics/items.rst

@@ -8,12 +8,21 @@ Items
    :synopsis: Item and Field classes
 
 The main goal in scraping is to extract structured data from unstructured
-sources, typically, web pages. Scrapy provides the :class:`Item` class for this
-purpose.
+sources, typically, web pages. Scrapy spiders can return the extracted data
+as Python dicts. While convenient and familiar, Python dicts lack structure:
+it is easy to make a typo in a field name or return inconsistent data,
+especially in a larger project with many spiders.
 
+To define common output data format Scrapy provides the :class:`Item` class.
 :class:`Item` objects are simple containers used to collect the scraped data.
 They provide a `dictionary-like`_ API with a convenient syntax for declaring
-their available fields.
+their available fields. 
+
+Various Scrapy components use extra information provided by Items: 
+exporters look at declared fields to figure out columns to export,
+serialization can be customized using Item fields metadata, :mod:`trackref`
+tracks Item instances to help finding memory leaks 
+(see :ref:`topics-leaks-trackrefs`_), etc.
 
 .. _dictionary-like: http://docs.python.org/library/stdtypes.html#dict
 
@@ -64,8 +73,6 @@ It's important to note that the :class:`Field` objects used to declare the item
 do not stay assigned as class attributes. Instead, they can be accessed through
 the :attr:`Item.fields` attribute.
 
-And that's all you need to know about declaring items.
-
 Working with Items
 ==================
 

docs/topics/practices.rst

@@ -190,11 +190,10 @@ Dynamic Creation of Item Classes
 ================================
 
 For applications in which the structure of item class is to be determined by
-user input, or other changing conditions, you can dynamically create item
-classes instead of manually coding them.
-
-::
+user input or other changing conditions you can return regular Python 
+dicts from spiders.
 
+Another option is to dynamically create Item classes::
 
     from scrapy.item import DictItem, Field
 

docs/topics/signals.rst

@@ -71,7 +71,7 @@ item_scraped
     This signal supports returning deferreds from their handlers.
 
     :param item: the item scraped
-    :type item: :class:`~scrapy.item.Item` object
+    :type item: dict or :class:`~scrapy.item.Item` object
 
     :param spider: the spider which scraped the item
     :type spider: :class:`~scrapy.spider.Spider` object
@@ -91,7 +91,7 @@ item_dropped
     This signal supports returning deferreds from their handlers.
 
     :param item: the item dropped from the :ref:`topics-item-pipeline`
-    :type item: :class:`~scrapy.item.Item` object
+    :type item: dict or :class:`~scrapy.item.Item` object
 
     :param spider: the spider which scraped the item
     :type spider: :class:`~scrapy.spider.Spider` object

docs/topics/spider-middleware.rst

@@ -90,15 +90,16 @@ following methods:
         it has processed the response.
 
         :meth:`process_spider_output` must return an iterable of
-        :class:`~scrapy.http.Request` or :class:`~scrapy.item.Item` objects.
+        :class:`~scrapy.http.Request`, dict or :class:`~scrapy.item.Item` 
+        objects.
 
         :param response: the response which generated this output from the
           spider
         :type response: :class:`~scrapy.http.Response` object
 
         :param result: the result returned by the spider
-        :type result: an iterable of :class:`~scrapy.http.Request` or
-          :class:`~scrapy.item.Item` objects
+        :type result: an iterable of :class:`~scrapy.http.Request`, dict
+          or :class:`~scrapy.item.Item` objects
 
         :param spider: the spider whose result is being processed
         :type spider: :class:`~scrapy.spider.Spider` object
@@ -110,7 +111,7 @@ following methods:
         method (from other spider middleware) raises an exception.
 
         :meth:`process_spider_exception` should return either ``None`` or an
-        iterable of :class:`~scrapy.http.Response` or
+        iterable of :class:`~scrapy.http.Response`, dict or
         :class:`~scrapy.item.Item` objects.
 
         If it returns ``None``, Scrapy will continue processing this exception,