diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst
index 673cda70d..e8bd7a50f 100644
--- a/docs/experimental/index.rst
+++ b/docs/experimental/index.rst
@@ -3,26 +3,21 @@
Experimental features
=====================
-This section documents experimental features that may become stable in the
-future, but whose API is not yet stable. Use them with caution, and subscribe
-to the `mailing lists `_ to be notified of any
-changes.
+This section documents experimental Scrapy features that may become stable in
+future releases, but whose API is not yet stable. Use them with caution, and
+subscribe to the `mailing lists `_ to get
+notified of any changes.
-This section may also contain documentation which is outdated or incomplete (as
-it's not revised so frequently), or documentation which overlaps with existing
-(stable) documentation and needs to be manually merged, use at your own risk.
+Since it's not revised so frequently, this section may contain documentation
+which is outdated, incomplete or overlapping with stable documentation (until
+it's properly merged) . Use at your own risk.
.. warning::
- This documentation is a work in progress, use at your own risk.
+ This documentation is a work in progress. Use at your own risk.
.. toctree::
:maxdepth: 1
- topics/index
-
-.. toctree::
- :maxdepth: 2
-
- ref/index
-
+ newitem
+ newitem-loader
diff --git a/docs/experimental/newitem-loader.rst b/docs/experimental/newitem-loader.rst
new file mode 100644
index 000000000..61d8d71fb
--- /dev/null
+++ b/docs/experimental/newitem-loader.rst
@@ -0,0 +1,357 @@
+.. _topics-loader:
+
+============
+Item Loaders
+============
+
+.. module:: scrapy.newitem.loader
+ :synopsis: Item Loader class
+
+Item Loaders (or Loaders, for short) provide a convenient mechanism for
+populating scraped :ref:`Items `. Even though Items can be
+populated using their own dictionary-like API, the Loaders provide a much more
+convenient API for populating them from a scraping process, by automating some
+common tasks like parsing the raw extracted data before assigning it.
+
+In other words, :ref:`Items ` provide the *container* of
+scraped data, while Loaders provide the mechanism for *populating* that
+container.
+
+Loaders are designed to provide a flexible, efficient and easy mechanism for
+extending and overriding different field parsing rules (either by spider, or by
+source format) without becoming a nightmare to maintain
+
+Using Loaders to populate items
+===============================
+
+To use a Loader, you must first instantiate it. You can either instantiate it
+with an Item object or without one, in which case an Item is automatically
+instantiated in the Loader constructor using the Item class specified in the
+:attr:`ItemLoader.default_item_class` attribute.
+
+Then, you start adding values to the Loader, typically collecting them using
+:ref:`Selectors `. You can add more than one value to the
+same item field, the Loader will know how to "join" those values later using a
+Reducer.
+
+Here's a typical Loader usage in a :ref:`Spider ` the
+:ref:`Product item defined in the Items chapter `.::
+
+ from scrapy.item.loader import Loader
+ from scrapy.xpath import HtmlXPathSelector
+ from myproject.items import Product
+
+ def parse(self, response):
+ x = HtmlXPathSelector(response)
+ l = ItemLoader(item=Product())
+ l.add_value('name', x.x('//div[@class="product_name"]').extract())
+ l.add_value('name', x.x('//div[@class="product_title"]').extract())
+ l.add_value('price', x.x('//p[@id="price"]').extract())
+ l.add_value('stock', x.x('//p[@id="stock"]').extract())
+ l.add_value('last_updated', 'today')
+ return l.get_item()
+
+By looking at that code we can see the ``name`` field is being extracted from
+two different XPath locations in the page:
+
+* ``//div[@class="product_name"]``
+* ``//div[@class="product_title"]``
+
+So both XPaths are used for extracting data from the page, and the data
+returned by them is collected to be assigned to the ``name`` attribute.
+
+Afterwards, similar calls are used for ``price`` and ``stock`` fields, and
+finally the ``last_update`` field is populated directly with a literal value
+(``today``).
+
+Finally, when all data is collected, the :meth:`ItemLoader.get_item` method is
+called which actually populates and returns the item populated with the data
+previously extracted with the ``add_value`` calls.
+
+Expanders and Reducers
+======================
+
+A Loader is composed of one expander and reducer for each item field. The
+Expander processes the extracted data as soon as it's received through the
+:meth:`ItemLoader.add_value` method, and the result of the expander is collected
+and kept inside the Loader. After collecting all data, the
+:meth:`ItemLoader.get_item` method is called to actually populate and get the Item.
+That's when the Reducers are called with the data previously collected (using
+the Expanders) and the output of the Reducers are the actual values that get
+assigned to the item.
+
+Let's see an example to illustrate how Expanders and Reducers are called, for a
+particular field (the same applies for any other field)::
+
+ l = ItemLoader(Product())
+ l.add_value('name', x.x(xpath1).extract()) # (1)
+ l.add_value('name', x.x(xpath2).extract()) # (2)
+ return l.get_item() # (3)
+
+So what happens is:
+
+1. Data from ``xpath1`` is extracted, and passed through the Expander of the
+ ``name`` field. The output of the expander is collected and kept in the
+ loader (but not yet assigned to the item).
+
+2. Data from ``xpath2`` is extracted, and passed through the same Expander used
+ in (1). The output of the expander is appended to the data collected in (1)
+ (if any).
+
+3. The data collected in (1) and (2) is passed through the Reducer of the
+ ``name`` field. The output of the Reducer is the value assigned to the
+ ``name`` field in the item.
+
+Scrapy comes with one major expander built-in, the :ref:`Tree Expander
+`, and :ref:`a couple of commonly used reducers
+`.
+
+Declaring Loaders
+=================
+
+Loaders are declared like Items, by using a class definition syntax. Here is an
+example::
+
+ from scrapy.newitem.loader import ItemLoader
+ from scrapy.newitem.loader.expanders import TreeExpander
+ from scrapy.newitem.loader.reducers import Join, TakeFirst
+
+ class ProductLoader(ItemLoader):
+
+ default_expander = TakeFirst()
+
+ name_exp = TreeExpander(unicode.title)
+ name_red = Join()
+
+ price_exp = TreeExpander(unicode.strip)
+ price_red = TakeFirst()
+
+ ...
+
+As you can see, expanders are declared using the ``_exp`` suffix while reducers
+are declared using the ``_red`` suffix. And you can also declare a default
+expander using the :attr:`ItemLoader.default_expander` attribute.
+
+.. _topics-loader-args:
+
+Item Loader arguments
+=====================
+
+The Loader arguments is a dict of arbitrary key/values which can be passed when
+declaring, instantiating or using Loaders. They are used modify the behaviour
+of the expanders.
+
+For example, suppose you have a function ``parse_length`` which receives a text
+value and extracts a length from it::
+
+ def parse_length(text, loader_args):
+ unit = loader_args('unit', 'm')
+ # ... length parsing code goes here ...
+ return parsed_length
+
+Since it receives a ``loader_args`` the Expander will pass the currently active
+loader arguments when calling it.
+
+There are seveal ways to pass loader arguments:
+
+1. Passing arguments on Loader declaration::
+
+ class ProductLoader(ItemLoader):
+ length_exp = TreeExpander(parse_length, unit='cm')
+
+2. Passing arguments on Loader instantiation::
+
+ l = ItemLoader(product, unit='cm')
+
+3. Passing arguments on Loader usage::
+
+ l.add_value('length', x.x('//div').extract(), unit='cm')
+
+ItemLoader objects
+==================
+
+.. class:: ItemLoader([item], \**loader_args)
+
+ Return a new Item Loader for populating the given Item. If no item is
+ given, one is instantiated using the class in :attr:`default_item_class`.
+
+ .. method:: add_value(field_name, value, \**new_loader_args)
+
+ Add the given ``value`` for the given field.
+
+ The value is passed through the field expander and its output appened
+ to the data collected for that field. If the field already contains
+ collected data, the new data is added.
+
+ If any keyword arguments are passed, they're used as :ref:`Loader
+ arguments ` when calling the expanders.
+
+ .. method:: replace_value(field_name, value, \**new_loader_args)
+
+ Similar to :meth:`add_value` but replaces collected data instead of
+ adding it.
+
+
+ .. method:: get_item()
+
+ Populate the item with the data collected so far, and return it.
+
+ .. method:: get_expanded_value(field_name)
+
+ Return the expanded data for the given field. In other words, return
+ the dat collected so far for the given field, without reducing it.
+
+ .. method:: get_reduced_value(field_name)
+
+ Return the reduced value for the given field, without modifying the
+ item.
+
+ .. method:: get_expander(field_name)
+
+ Return the expander for the given field.
+
+ .. method:: get_reducer(field_name)
+
+ Return the reducer for the given field.
+
+ .. attribute:: default_item_class
+
+ An Item class (or factory), used to instantiate items when not given in
+ the constructor.
+
+ .. attribute:: default_expander
+
+ The default expander to use for those fields which don't define a
+ specific expander
+
+ .. attribute:: default_reducer
+
+ The default reducer to use for those fields which don't define a
+ specific expander
+
+Reusing and extending Loaders
+=============================
+
+As your project grows bigger and acquires more and more spiders, maintenance
+becomes a fundamental problem, specially when you have to deal with many
+different parsing rules per spider, a lot of exceptions, but also want to reuse
+the common cases.
+
+Loaders are designed to ease the maintenance of parsing rules, without loosing
+flexibility and, at the same time, providing a convenient mechanism for
+extending and overriding them. For this reason Loaders support traditional
+class inheritance for for dealing with differences of specific spiders (or
+group of spiders).
+
+Suppose, for example, that some particular site encloses their product names
+between three dashes (ie. ``---Plasma TV---``) and you don't want to end up
+scraping those dashes in the final product names.
+
+Here's how you can remove those dashes by reusing and extending the default
+Product Loader::
+
+ strip_dashes = lambda x: x.strip('-')
+
+ class SiteSpecificLoader(ProductLoader):
+ name_exp = TreeExpander(ProductLoader.name_exp, strip_dashes)
+
+Another case where extending Loaders can be very helpful is when you have
+multiple source formats, for example XML and HTML. In the XML version you may
+want to remove ``CDATA`` occurrences. Here's an example of how to do it::
+
+ from myproject.utils.xml import remove_cdata
+
+ class XmlLoader(ProductLoader):
+ name_exp = TreeExpander(remove_cdata, ProductLoader.name_exp)
+
+There are many other possible ways to extend, inherit and override your
+Loaders, and different Loader hierarchies may fit better for different
+projects. Scrapy only provides the mechanism, it doesn't impose any specific
+organization of your Loaders collection - that's up to you and your project
+needs.
+
+Available Expanders
+===================
+
+.. _topics-loader-tree-expander:
+
+Tree Expander
+-------------
+
+The Tree Expander is the recommended Expander to use and the only really useful
+one, as the other is just an identity expander.
+
+.. module:: scrapy.newitem.loader.expanders
+ :synopsis: Expander classes to use with Item Loaders
+
+.. class:: TreeExpander(\*functions, \**default_loader_arguments)
+
+ An expander which applies the given functions consecutively, in order, to
+ each value returned by the previous function.
+
+ The algorithm consists in an ordered list of functions, each of which
+ receives one value and can return zero, one or more values (as a list or
+ iterable). If a function returns more than one value, the next function in
+ the list will be called with each of those values, potentially returning
+ more values and thus expanding the execution into different branches, which
+ is why this expander is called Tree Expander.
+
+ Each expander function can optionally receive a ``loader_args`` argument,
+ which will contain the currently active loader arguments.
+
+ The keyword arguments passed in the consturctor are used as the default
+ loader arguments passed to on each expander call. This arguments can be
+ overriden with specific loader arguments passed on each expander call.
+
+IdentityExpander
+----------------
+
+.. class:: IdentityExpander
+
+ An expander which returns the original values unchanged. It doesn't support
+ any constructor arguments.
+
+.. _topics-loader-reducers:
+
+Available Reducers
+==================
+
+.. module:: scrapy.newitem.loader.reducers
+ :synopsis: Reducer classes to use with Item Loaders
+
+Reducers are callable objects which are called with a list of values (to be
+reduced) as their first and only argument. Scrapy provides some simple,
+commonly used reducers, which are described below. But you can use any function
+or callable as reducer.
+
+.. class:: TakeFirst
+
+ Return the first non-null value from the values to reduce, so it's used for
+ single-valued fields. It doesn't receive any constructor arguments.
+
+ Example::
+
+ name_red = TakeFirst()
+
+.. class:: Identity
+
+ Return the values to reduce unchanged, so it's used for multi-valued
+ fields. It doesn't receive any constructor arguments.
+
+ Example::
+
+ features_red = Identity()
+
+.. class:: Join(separator=u' ')
+
+ Return a the values to reduce joined with the separator given in the
+ constructor, which defaults to ``u' '``.
+
+ When using the default separator, this reducer is equivalent to the
+ function: ``u' '.join``
+
+ Examples::
+
+ name_red = Join()
+ name_red = Join('
')
+
diff --git a/docs/experimental/newitem.rst b/docs/experimental/newitem.rst
new file mode 100644
index 000000000..ec5c2149e
--- /dev/null
+++ b/docs/experimental/newitem.rst
@@ -0,0 +1,239 @@
+.. _topics-newitem:
+
+=====
+Items
+=====
+
+.. module:: scrapy.newitem
+ :synopsis: Item and Field classes
+
+The main goal in scraping is to extract structured data from unstructured
+sources, typically, web pages. Scrapy provides the :class:`Item` class for this
+purpose.
+
+:class:`Item` objects are simple containers used to collect the scraped data.
+They provide a `dictionary-like`_ API with a convenient syntax for declaring
+their available fields.
+
+.. _dictionary-like: http://docs.python.org/library/stdtypes.html#dict
+
+.. _topics-newitem-declaring:
+
+Declaring Items
+===============
+
+Items are declared using a simple class definition syntax and :class:`Field`
+objects. Here is an example::
+
+ from scrapy.newitem import Item, Field
+
+ class Product(Item):
+ name = Field()
+ price = Field()
+ stock = Field(default=0)
+ last_updated = Field()
+
+.. note:: Those familiar with `Django`_ will notice that Scrapy Items are
+ declared similar to `Django Models`_, except that Scrapy Items are much
+ simpler as there is no concept of different field types.
+
+.. _Django: http://www.djangoproject.com/
+.. _Django Models: http://docs.djangoproject.com/en/dev/topics/db/models/
+
+:class:`Field` objects are used to specify metadata for each field. For
+example, the default value for the ``stock`` field illustrated in the example
+above.
+
+You can specify any kind of metadata for each field. There is no restriction on
+the values accepted by :class:`Field` objects. For this same
+reason, there isn't a reference list of all available metadata keys. Each key
+defined in :class:`Field` objects could be used by a different components, and
+only those components know about it. You can also define and use any other
+:class:`Field` key in your project too, for your own needs. The main goal of
+:class:`Field` objects is to provide a way to define all field metadata in one
+place. Typically, those components whose behaviour depends on each field, use
+certain field keys to configure that behaviour. You must refer to their
+documentation to see which metadata keys are used by each component.
+
+It's important to note that the :class:`Field` objects used to declare the item
+do not stay assigned as class attributes. Instead, they can be accesed through
+the :attr:`Item.fields` attribute.
+
+And that's all you need to know about declaring items.
+
+Working with Items
+==================
+
+Here are some examples of common tasks performed with items, using the
+``Product`` item :ref:`declared above `. You will
+notice the API is very similar to the `dict API`_.
+
+Creating items
+--------------
+
+::
+
+ >>> product = Product(name='Desktop PC', price=1000)
+ >>> print product
+ Product(name='Desktop PC', price=1000)
+
+Getting field values
+--------------------
+
+::
+
+ >>> product['name']
+ Desktop PC
+ >>> product.get('name')
+ Desktop PC
+
+ >>> product['price']
+ 1000
+
+ >>> product['stock'] # getting field with default value
+ 0
+
+ >>> product['last_updated'] # getting field with no default value
+ Traceback (most recent call last):
+ ...
+ KeyError: 'last_updated'
+
+ >>> product.get('last_updated', 'not set')
+ not set
+
+ >>> product['lala'] # getting unknown field
+ Traceback (most recent call last):
+ ...
+ KeyError: 'lala'
+
+ >>> product.get('lala', 'unknown field')
+ 'unknown field'
+
+ >>> 'name' in product # is name field populated?
+ True
+
+ >>> 'last_updated' in product # is last_updated populated?
+ False
+
+ >>> 'last_updated' in product.fields # is last_updated a declared field?
+ True
+
+ >>> 'lala' in product.fields # is lala a declared field?
+ False
+
+Setting field values
+--------------------
+
+::
+
+ >>> product['last_updated'] = 'today'
+ >>> product['last_updated']
+ today
+
+ >>> product['lala'] = 'test' # setting unknown field
+ Traceback (most recent call last):
+ ...
+ KeyError: 'Product does not support field: lala'
+
+Accesing all populated values
+-----------------------------
+
+To access all populated values just use the typical `dict API`_::
+
+ >>> product.keys()
+ ['price', 'name']
+
+ >>> product.items()
+ [('price', 1000), ('name', 'Desktop PC')]
+
+Other common tasks
+------------------
+
+Copying items::
+
+ >>> product2 = Product(product)
+ >>> print product2
+ Product(name='Desktop PC', price=1000)
+
+Creating dicts from items::
+
+ >>> dict(product) # create a dict from all populated values
+ {'price': 1000, 'name': 'Desktop PC'}
+
+Creating items from dicts::
+
+ >>> Product({'name': 'Laptop PC', 'price': 1500})
+ Product(price=1500, name='Laptop PC')
+
+ >>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict
+ Traceback (most recent call last):
+ ...
+ KeyError: 'Product does not support field: lala'
+
+Default values
+==============
+
+The only field metadata key supported by Items themselves is ``default``, which
+specifies the default value to return when trying to access a field which
+wasn't populated before.
+
+So, for the ``Product`` item declared above::
+
+ >>> product = Product()
+
+ >>> product['stock'] # field with default value
+ 0
+
+ >>> product['name'] # field with no default value
+ Traceback (most recent call last):
+ ...
+ KeyError: 'name'
+
+ >>> product.get('name') is None
+ True
+
+Extending Items
+===============
+
+You can extend Items (to add more fields or to change some metadata for some
+fields) by declaring a subclass of your original Item.
+
+For example::
+
+ class DiscountedProduct(Product):
+ discount_percent = Field(default=0)
+ discount_expiration_date = Field()
+
+Item objects
+============
+
+.. class:: Item([arg])
+
+ Return a new Item optionally initialized from the given argument.
+
+ Items replicate the standard `dict API`_, including its constructor. The
+ only additional attribute provided by Items is:
+
+ .. attribute:: fields
+
+ A dictionary containing *all declared fields* for this Item, not only
+ those populated. The keys are the field names and the values are the
+ :class:`Field` objects used in the :ref:`Item declaration
+ `.
+
+.. _dict API: http://docs.python.org/library/stdtypes.html#dict
+
+Field objects
+=============
+
+.. class:: Field([arg])
+
+ The :class:`Field` class is just an alias to the built-in `dict`_ class and
+ doesn't provide any extra functionality or attributes. In other words,
+ :class:`Field` objects are plain-old Python dicts. A separate class is used
+ to support the :ref:`item declaration syntax `
+ based on class attributes.
+
+.. _dict: http://docs.python.org/library/stdtypes.html#dict
+
+
diff --git a/docs/experimental/ref/index.rst b/docs/experimental/ref/index.rst
deleted file mode 100644
index 9ef0d55cf..000000000
--- a/docs/experimental/ref/index.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-.. toctree::
-
- newitem/index
diff --git a/docs/experimental/ref/newitem/fields.rst b/docs/experimental/ref/newitem/fields.rst
deleted file mode 100644
index dd2e4b5e5..000000000
--- a/docs/experimental/ref/newitem/fields.rst
+++ /dev/null
@@ -1,200 +0,0 @@
-.. _ref-newitem-fields:
-
-===========
-Item Fields
-===========
-
-.. module:: scrapy.newitem.fields
-
-
-Field options
-=============
-
-Every ``Field`` class constructor accepts these arguments.
-
-
-``default``
------------
-
-The default value for the field. See :ref:`topics-newitem-index-defaults`.
-
-
-Field types
-===========
-
-These are the available built-in ``Field`` types. See
-:ref:`ref-newitem-fields-custom-fields` for info on creating your own field types.
-
-
-``BooleanField``
-----------------
-
-.. class:: BooleanField
-
- A boolean (true/false) field.
-
-
-``DateField``
--------------
-
-.. class:: DateField
-
- A date, represented in Python by a `datetime.date`_ instance.
-
-.. _datetime.date: http://docs.python.org/library/datetime.html#datetime.date
-
-
-``DateTimeField``
------------------
-
-.. class:: DateTimeField
-
- A date with time, represented in Python by a `datetime.datetime`_ instance.
-
-.. _datetime.datetime: http://docs.python.org/library/datetime.html#datetime.datetime
-
-
-``DecimalField``
-----------------
-
-.. class:: DecimalField
-
- A fixed-precision decimal number, represented in Python by a `Decimal`_
- instance.
-
-.. _Decimal: http://docs.python.org/library/decimal.html#decimal.Decimal
-
-
-``FloatField``
---------------
-
-.. class:: FloatField
-
- A floating-point number represented in Python by a ``float`` instance.
-
-
-``IntegerField``
-----------------
-
-.. class:: IntegerField
-
- An integer.
-
-
-``ListField``
--------------
-
-.. class:: ListField(field)
-
- A special field that works like a list of fields of another provided field kind.
-
- :param field: The field which the elements of this list must conform to.
- :type field: a :class:`~scrapy.newitem.fields.BaseField` object
-
- Usage example::
-
- class ExampleItem(Item)
- names = fields.ListField(fields.TextField())
-
- item = ExampleItem()
- item['names'] = [u'John', u'Jeena']
-
-
-``TextField``
--------------
-
-.. class:: TextField
-
- A unicode text.
-
- This class overrides the following methods from :class:`BaseField`:
-
- .. method:: from_unicode_list(unicode_list)
-
- Return a unicode string composed by joining the elements of
- ``unicode_list`` with spaces.
-
- For more info about this method see :class:`BaseField.from_unicode_list`.
-
-
-``TimeField``
--------------
-
-.. class:: TimeField
-
- A time, represented in Python by a `datetime.time`_ instance.
-
-.. _datetime.time: http://docs.python.org/library/datetime.html#datetime.time
-
-.. _ref-newitem-fields-custom-fields:
-
-
-Creating custom fields
-======================
-
-All field classes are subclasses of the :class:`BaseField` class (see below)
-which you can also subclass to create your own custom fields.
-
-You can also subclass a more specific field class, say :class:`DecimalField`,
-to implement a ``PriceField``, for example.
-
-
-BaseField class
----------------
-
-.. class:: BaseField(default=None)
-
- The base class for all fields. It only provides code for handling default
- values, not any particular type. It cannot be used directly either, as its
- :meth:`BaseField.to_python` method is not implemented.
-
- The ``default`` argument (if given) must be of the type expected by this
- field, or any type that is accepted by the :meth:`BaseField.to_python`
- method of this field.
-
- For example::
-
- class NewsItem(Item):
- content = fields.TextField() # correct, no default value
- author = fields.TextField(default=u'Myself") # correct, with default value
- published = fields.DateField(default=23) # wrong default type (will raise TypeError)
-
- .. method:: to_python(value)
-
- Convert the input value to the type expected by this field and return
- it.
-
- For example, :class:`IntegerField` would convert ``'1'`` to ``1``, while
- :class:`DecimalField` would convert ``'1'`` to ``Decimal('1')`` and so
- on.
-
- This method is not implemented in the :class:`BaseField` class, so it
- must always be implemented in all its subclasses, in order to be usable.
-
- This method should raise ``TypeError`` if the input type is not
- supported, and ``ValueError`` if the input type is support but its value
- is not appropriate (for example, an integer outside a given range).
-
- This method must always return object of the expected field type.
-
- .. method:: from_unicode_list(unicode_list)
-
- Take the input list of unicode strings and convert it to a proper value
- with the type expected by this field. If no proper value if found,
- ``None`` is returned instead.
-
- The default behaviour is to return the value of the first item of the
- list, passed through the :meth:`to_python` method, or ``None`` if the
- list is empty::
-
- return self.to_python(unicode_list[0]) if unicode_list else None
-
- This default behaviour is provided because it's the more common one, but
- it's typical for :class:`BaseField` subclasses to override this method,
- such as the :meth:`TextField.from_unicode_list` method.
-
- .. method:: get_default()
-
- Return the default value for this field, or ``None`` if the field
- doesn't specify any.
-
diff --git a/docs/experimental/ref/newitem/index.rst b/docs/experimental/ref/newitem/index.rst
deleted file mode 100644
index 1e18c730d..000000000
--- a/docs/experimental/ref/newitem/index.rst
+++ /dev/null
@@ -1,12 +0,0 @@
-.. _ref-newitems-index:
-
-Items Reference
-===============
-
-This is the Items reference, for introductory material see :ref:`topics-newitem-index`
-
-.. toctree::
- :maxdepth: 1
-
- fields
-
diff --git a/docs/experimental/topics/index.rst b/docs/experimental/topics/index.rst
deleted file mode 100644
index 6c4ce6f27..000000000
--- a/docs/experimental/topics/index.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-.. toctree::
-
- newitem/index
- newitem/adaptors
diff --git a/docs/experimental/topics/newitem/adaptors.rst b/docs/experimental/topics/newitem/adaptors.rst
deleted file mode 100644
index b4448de06..000000000
--- a/docs/experimental/topics/newitem/adaptors.rst
+++ /dev/null
@@ -1,139 +0,0 @@
-.. _topics-newitem-adaptors:
-
-=============
-Item Adaptors
-=============
-
-.. class:: scrapy.contrib_exp.newitem.adaptors.ItemAdaptor
-
-As you probably want to scrape the same kind of Items from many sources
-(different websites, RSS feeds, etc.), Scrapy implements ItemAdaptors, they
-allow you to adapt chunks of HTML or XML (selected using Selectors) to the
-expected format of your Item fields.
-
-An ItemAdaptor acts like a wrapper of an Item, you define an ItemAdaptor class,
-set the Item class to wrap and assign a set of functions (adaptor functions) to be called when you assign a value to a field.
-
-Here's an example of an ItemAdaptor for our previously created Item::
-
- class NewsAdaptor(ItemAdaptor):
- item_class = NewsItem
-
- url = adaptor(extract, remove_tags(), unquote(), strip)
- headline = adaptor(extract, remove_tags(), unquote(), strip)
- summary = adaptor(extract, remove_tags(), unquote(), strip)
- content = adaptor(extract, remove_tags(), unquote(), strip)
-
-How do we use it? Let's see it in action in a Spider::
-
- def parse_newspage(self, response):
- xhs = HtmlXPathSelector(response)
- i = NewsAdaptor(response)
-
- i.url = response.url
- i.headline = xhs.x('//h1[@class="headline"]')
- i.summary = xhs.x('//div[@class="summary"]')
- i.content = xhs.x('//div[@id="body"]')
- # published attribute is intentionally omitted, see below for site-specific adaptors
- return [i]
-
-What happens underneath?
-
-When we assign a value to a ItemAdaptor field it passes for the chain of
-functions defined previously in it's class, in this case, the value gets
-extracted (note that we assign directly the value obtained from the Selector),
-then tags will be removed, then the result will be unquoted, stripped and
-finally assigned to the Item Field.
-
-This final assignment is done in an internal instance of the Item on the
-ItemAdaptor, that's why we can return an ItemAdaptor instead of an Item and
-Scrapy will know how to extract the item from it.
-
-An Item can have as many ItemAdaptors as you want. It generally depends on how
-many sources and formats are you scraping from.
-
-ItemAdaptor inheritance
-=======================
-
-As we said before you generally want an ItemAdaptor for each different source of
-data and maybe some for specific sites, inheritance make this really easy, let's
-see an example of adapting HTML and XML::
-
- class NewsAdaptor(ItemAdaptor):
- item_class = NewsItem
-
-
- class HtmlNewsAdaptor(NewsAdaptor):
- url = adaptor(extract, remove_tags(), unquote(), strip)
- headline = adaptor(extract, remove_tags(), unquote(), strip)
- summary = adaptor(extract, remove_tags(), unquote(), strip)
- content = adaptor(extract, remove_tags(), unquote(), strip)
- published = adaptor(extract, remove_tags(), unquote(), strip)
-
-
- class XmlNewsAdaptor(HtmlNewsAdaptor):
- url = adaptor(extract, remove_root, strip)
- headline = adaptor(extract, remove_root, strip)
- summary = adaptor(extract, remove_root, strip)
- content = adaptor(extract, remove_root, strip)
- published = adaptor(extract, remove_root, strip)
-
-
-Site specific ItemAdaptors
-==========================
-
-For the moment we have covered adapting information from different sources, but
-other common case is adapting information for specific sites, think for example
-in our published field, it keeps the publication date of the news article.
-
-As sites offer this information in different formats, we will have to make
-custom adaptors for it, let's see an example using our Item published field::
-
- class SpecificSiteNewsAdaptor(HtmlNewsAdaptor):
- published = adaptor(HtmlNewsAdaptor.published, to_date('%d.%m.%Y'))
-
-
-The ``to_date`` adaptor function converts a string with the format specified in
-its parameter to one in 'YYYY-mm-dd' format (the one that DateField expects).
-
-And in this example we're appending it to the of the chain of adaptor functions
-of published.
-
-Note that ``SpecificSiteNewsAdaptor`` will inherit the field adaptations from
-``HtmlNewsAdaptor``.
-
-Let's see it in action::
-
- def parse_newspage(self, response):
- xhs = HtmlXPathSelector(response)
- i = SpecificSiteNewsAdaptor(response)
-
- i.url = response.url
- i.headline = xhs.x('//h1[@class="headline"]')
- i.summary = xhs.x('//div[@class="summary"]')
- i.content = xhs.x('//div[@id="body"]')
- i.published = xhs.x('//h1[@class="date"]').re('\d{2}\.\d{2}\.\d{4}')
- return [i]
-
-ItemAdaptor default_adaptor
-===========================
-
-If you look closely at the code for our ItemAdaptors you can see that we're
-using the same set of adaptation functions in every field.
-
-It is common for ItemAdaptors to have a basic set of adaptor functions that
-will be applied to almost every Field in the Item. To avoid repeating the same
-code, ItemAdaptor implements the ``default_adaptor`` shortcut.
-
-``default_adaptor`` (if set) will be called when assigning a value for an Item
-Field that has no adaptor, so the process for determining what value gets
-assigned to an item when you assign a value to an ItemAdaptor field is as
-follows:
-
-1. If there's an adaptor function for this field its called before assigning
- the value to the item.
-2. If no adaptor function if set and default_adaptor is, the value passes for
- ``default_adaptor`` before being assigned.
-3. If no adaptor is defined for that field and no ``default_adaptor`` is set,
- the value is assigned directly.
-
diff --git a/docs/experimental/topics/newitem/index.rst b/docs/experimental/topics/newitem/index.rst
deleted file mode 100644
index 976d393ea..000000000
--- a/docs/experimental/topics/newitem/index.rst
+++ /dev/null
@@ -1,103 +0,0 @@
-.. _topics-newitem-index:
-
-.. _topics-newitem-scrapeditem:
-
-=====
-Items
-=====
-
-.. currentmodule:: scrapy.item
-
-The goal of the scraping process is to obtain scraped items from scraped pages.
-
-Basic scraped items
-===================
-
-In Scrapy the items are represented by a :class:`ScrapedItem` (almost an empty
-class) or any subclass of it.
-
-To use :class:`ScrapedItem` you simply instantiate it and use instance
-attributes to store the information.
-
- >>> from scrapy.item import ScrapedItem
- >>> item = ScrapedItem()
- >>> item.headline = 'Headline'
- >>> item.content = 'Content'
- >>> item.published = '2009-07-08'
- >>> item
- ScrapedItem({'headline': 'Headline', 'content': 'Content', 'published': '2009-07-08'})
-
-Or you can use your own class to represent items, just be sure it inherits from
-:class:`ScrapedItem`.
-
-.. _topics-newitem-index-item:
-
-More advanced items
-===================
-
-.. currentmodule:: scrapy.newitem
-
-Scrapy provides :class:`Item` (a subclass of :class:`~scrapy.item.ScrapedItem`)
-that works like a form with fields to store the item's data.
-
-To use this items you first define the item's fields as class attributes::
-
- from scrapy.newitem import Item
- from scrapy.newitem import fields
-
- class NewsItem(Item):
- headline = fields.TextField()
- content = fields.TextField()
- published = fields.DateField()
-
-And then you instantiate the item and assign values to its fields, which will be
-converted to the expected Python types depending of their class::
-
- >>> item = NewsItem()
- >>> item['headline'] = u'Headline'
- >>> item['content'] = u'Content'
- >>> item['published'] = '2009-07-08'
- >>> item
- NewsItem(headline=u'Headline', content=u'Content', published=datetime.date(2009, 7, 8))
-
-Using this may seen complicated at first, but gives you much power over scraped
-data, like :ref:`topics-newitem-index-defaults`,
-:ref:`topics-newitem-adaptors`, etc.
-
-.. _topics-newitem-index-defaults:
-
-Default values for fields
--------------------------
-
-Each field accepts a ``default`` argument, that sets the default value of the
-field.
-
-Fields which contain a default value will always return that value when not
-set, while fields which don't contain a default value will raise ``KeyError``,
-you can use :meth:`Item.get` method to avoid this.
-
-.. code-block:: python
-
- from scrapy.newitem import Item, fields
-
- class NewsItem(Item):
- headline = fields.TextField()
- content = fields.TextField()
- published = fields.DateField()
- author = fields.TextField(default=u'Myself')
- views = fields.IntegerField(default=0)
-
-.. code-block:: python
-
- >>> item = NewsItem()
- >>> item['author']
- u'Myself'
- >>> item['views']
- 0
- >>> item['headline']
- Traceback (most recent call last):
- ...
- KeyError: 'headline
- >>> item.get('headline') is None
- True
-
diff --git a/docs/faq.rst b/docs/faq.rst
index cce5a811c..8a1fd8667 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -51,7 +51,7 @@ to steal from us!
Does Scrapy work with HTTP proxies?
-----------------------------------
-No. support for HTTP proxies is not currently implemented in Scrapy, but it
+No. Support for HTTP proxies is not currently implemented in Scrapy, but it
will be in the future. For more information about this, follow `this ticket
`_. Setting the ``http_proxy`` environment
variable won't work because Twisted (the library used by Scrapy to download
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index ceb8cb0c7..4fbfe2870 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -57,7 +57,7 @@ RobustScrapedItems
.. warning::
RobustScapedItems are deprecated and will be replaced by the :ref:`New item
- API ` (still in development).
+ API ` (still in development).
.. module:: scrapy.contrib.item
:synopsis: Objects for storing scraped data