6.4 KiB
Items
Quick overview
In Scrapy, items are the placeholder to use for the scraped data. They are represented by a :class:`ScrapedItem` object, or any descendant class instance, and store the information in instance attributes.
System Message: ERROR/3 (<stdin>, line 10); backlink
Unknown interpreted text role "class".ScrapedItems
System Message: ERROR/3 (<stdin>, line 17)
Unknown directive type "module".
.. module:: scrapy.item :synopsis: Objects for storing scraped data
Methods
System Message: ERROR/3 (<stdin>, line 25)
Unknown directive type "method".
.. method:: ScrapedItem.__init__(data=None)
:param data: A dictionary containing attributes and values to be set
after instancing the item.
Instanciates a ``ScrapedItem`` object and sets an attribute and its value
for each key in the given ``data`` dict (if any). These items are the most
basic items available, and the common interface from which any items should
inherit.
Examples
Creating an item and setting some attributes:
>>> from scrapy.item import ScrapedItem
>>> item = ScrapedItem()
>>> item.name = 'John'
>>> item.last_name = 'Smith'
>>> item.age = 23
>>> item
ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'})
Creating an item and setting its attributes inline:
>>> person = ScrapedItem({'name': 'John', 'age': 23, 'last_name': 'Smith'})
>>> person
ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'})
RobustScrapedItems
System Message: ERROR/3 (<stdin>, line 57)
Unknown directive type "module".
.. module:: scrapy.contrib.item :synopsis: Objects for storing scraped data
RobustScrapedItems are more complex items (compared to :class:`ScrapedItem`) and have a few more features available, which include:
System Message: ERROR/3 (<stdin>, line 62); backlink
Unknown interpreted text role "class".Attributes dictionary: items that inherit from RobustScrapedItem are defined with a dictionary of attributes in the class. This allows the item to have more logic at the moment of handling and setting attributes than the :class:`ScrapedItem`.
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "class".
Adaptors: perhaps the most important of the features these items provide. The adaptors are a system designed for filtering/modifying data before setting it to the item, that makes cleansing tasks a lot easier.
Type checking: RobustScrapedItems come with a built-in type checking which assures you that no data of the wrong type will get into the items without raising a warning.
Versioning: These items also provide versioning by making a unique hash for each item based on its attributes values.
ItemDeltas: You can subtract two RobustScrapedItems, which allows you to know the difference between a pair of items. This difference is represented by a RobustItemDelta object.
Attributes
System Message: ERROR/3 (<stdin>, line 89)
Unknown directive type "attribute".
.. attribute:: RobustScrapedItem.ATTRIBUTES
This attribute **must** be specified when writing your items, and it's a
dictionary in which the keys are the names of the attributes your item will
have, and their values are the type of those attributes. For multivalued
attributes, you should write the type of the values inside a list, e.g:
``'numbers': [int]``
Methods
System Message: ERROR/3 (<stdin>, line 100)
Unknown directive type "method".
.. method:: RobustScrapedItem.__init__(data=None, adaptor_args=None)
:param data: Idem as for ScrapedItems
:param adaptor_args: A dictionary of the kind
``'attribute': [list_of_adaptors]``" for defining adaptors automatically
after instancing the item.
Constructor of RobustScrapedItem objects.
System Message: ERROR/3 (<stdin>, line 109)
Unknown directive type "method".
.. method:: RobustScrapedItem.attribute(attrname, value, override=False, add=False, ***kwargs)
Sets the item's ``attrname`` attribute with the given ``value`` filtering
it through the given attribute's adaptor pipeline (if any).
:param attrname: a string containing the name of the attribute you want
to set.
:param value: the value you want to assign, which will be adapted by
the corresponding adaptors for the given attribute (if any).
:param override: if True, makes this method avoid checking if there
was a previous value and sets ``value`` no matter what.
:param add: if True, tries to concatenate the given ``value`` with the one
already set in the item. For multivalued attributes, this will extend
the list of already-set values, with the new ones.
For single valued attributes, the method _add_single_attributes (which
is explained below) will be called.
:param kwargs: any extra parameters will be passed in a dictionary to any
adaptor that receives a parameter called ``adaptor_args``.
Check the :ref:`topics-adaptors` topic for more information.
System Message: ERROR/3 (<stdin>, line 133)
Unknown directive type "method".
.. method:: RobustScrapedItem.set_adaptors(adaptors_dict)
Receives a dict containing a list of adaptors for each desired attribute
(key) and sets each of them as their adaptor pipeline.
System Message: ERROR/3 (<stdin>, line 138)
Unknown directive type "method".
.. method:: RobustScrapedItem.set_attrib_adaptors(attrib, pipe)
Sets the provided iterable (``pipe``) as the adaptor pipeline for the
given attribute (``attrib``)
System Message: ERROR/3 (<stdin>, line 143)
Unknown directive type "method".
.. method:: RobustScrapedItem.add_adaptor(attrib, adaptor, position=None)
Adds an adaptor to an already existing (or not) pipeline.
:param attr: the name of the attribute you're adding adaptors to.
:param adaptor: a callable to be added to the pipeline.
:param position: an integer representing the place where to add the adaptor.
If it's ``None``, the adaptor will be appended at the end of the pipeline.
System Message: ERROR/3 (<stdin>, line 154)
Unknown directive type "method".
.. method:: RobustScrapedItem._add_single_attributes(attrname, attrtype, attributes)
This method is the one to be called whenever a single attribute has to be
joined before storing into an item. That is,
every time you have multiple results at the end of your adaptors pipeline,
and you called the ``attribute`` method with the parameter `add=True`.
This method is intended to be overriden by you, since by default it
raises an exception.
:param attrname: the name of the attribute you're setting
:param attrtype: the type of the attribute you're setting
:param attributes: the list of resulting values after the adaptors pipeline
(the one you have to join somehow)
Examples
Creating a pretty basic item with a few attributes:
from scrapy.contrib.item import RobustScrapedItem
class MyItem(RobustScrapedItem):
ATTRIBUTES = {
'name': basestring,
'size': basestring,
'colours': [basestring],
}
Setting some adaptors:
.. note::
More RobustScrapedItem examples are about to come. In the meantime, check the :ref:`topics-adaptors` topic to see a few of them.
System Message: ERROR/3 (<stdin>, line 188); backlink
Unknown interpreted text role "ref".