6.1 KiB
New Items
The goal of the scraping process is to obtain scraped items from scraped pages.
Basic scraped items
In Scrapy the items are represented by a :class:`~scrapy.item.ScrapedItem` (almost an empty class) or any subclass of it.
System Message: ERROR/3 (<stdin>, line 12); backlink
Unknown interpreted text role "class".To use :class:`~scrapy.item.ScrapedItem` you simply instantiate it and use instance attributes to store the information.
System Message: ERROR/3 (<stdin>, line 15); backlink
Unknown interpreted text role "class".
>>> from scrapy.item import ScrapedItem
>>> item = ScrapedItem()
>>> item.headline = 'Headline'
>>> item.content = 'Content'
>>> item.published = '2009-07-08'
>>> item
ScrapedItem({'headline': 'Headline', 'content': 'Content', 'published': '2009-07-08'})
Or you can use your own class to represent items, just be sure it inherits from :class:`~scrapy.item.ScrapedItem`.
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "class".More advanced items
Scrapy provides :class:`~scrapy.contrib_exp.newitem.Item` (a subclass of :class:`~scrapy.item.ScrapedItem`) that works like a form with fields to store the item's data.
System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "class".To use this items you first define the item's fields as class attributes:
from scrapy.contrib_exp.newitem import Item
from scrapy.contrib_exp.newitem import fields
class NewsItem(Item):
headline = fields.TextField()
content = fields.TextField()
published = fields.DateField()
And then you instantiate the item and assign values to its fields, which will be converted to the expected Python types depending of their class:
>>> item = NewsItem()
>>> item.headline = u'Headline'
>>> item.content = u'Content'
>>> item.published = '2009-07-08'
>>> item
NewsItem({'headline': u'Headline', 'content': u'Content', 'published': datetime.date(2009, 7, 8)})
Each field accepts a default argument, that sets the default value of the field.
Using this may seen complicated at first, but gives you much power over scraped data, like assigning defaults for fields that are not present in some pages, :ref:`topic-newitem-adaptors`, etc.
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "ref".Item Fields
System Message: ERROR/3 (<stdin>, line 70)
Unknown directive type "module".
.. module:: scrapy.contrib_exp.newitem.fields
Field options
Every Field class constructor accepts these arguments.
default
The default value for the field.
Fields which contain a default value will always return that value when not set, while fields which don't contain a default value will always return None when not set:
from scrapy.contrib_exp.newitem import Item, fields
class NewsItem(Item):
content = fields.TextField()
author = fields.TextField(default=u'Myself')
published = fields.DateField()
views = fields.IntegerField(default=0)
>>> it = NewsItem()
>>> it.content is None
True
>>> it.author
u'Myself'
>>> it.published is None
True
>>> it.views
0
Field types
These are the available built-in Field types. See :ref:`newitem-custom-fields` for info on creating your own field types.
System Message: ERROR/3 (<stdin>, line 107); backlink
Unknown interpreted text role "ref".TextField
A unicode text.
IntegerField
An integer.
DecimalField
A fixed-precision decimal number, represented in Python by a Decimal instance.
FloatField
A floating-point number represented in Python by a float instance.
BooleanField
A boolean (true/false) field.
DateTimeField
A date with time, represented in Python by a datetime.datetime instance.
DateField
A date, represented in Python by a datetime.date instance.
TimeField
A time, represented in Python by a datetime.time instance.
Creating custom fields
All field classes are subclasses of the :class:`BaseField` class (see below) which you can also subclass to create your own custom fields.
System Message: ERROR/3 (<stdin>, line 181); backlink
Unknown interpreted text role "class".You can also subclass a more specific field class, say :class:`DecimalField`, to implement a PriceField, for example.
System Message: ERROR/3 (<stdin>, line 184); backlink
Unknown interpreted text role "class".BaseField class
The base class for all fields. It only provides code for handling default values, not any particular type. It cannot be used directly either, as its :meth:`BaseField.to_python` method is not implemented.
System Message: ERROR/3 (<stdin>, line 192); backlink
Unknown interpreted text role "meth".The default argument (if given) must be of the type expected by this field, or any type that is accepted by the :meth:BaseField.to_python method of this field.
For example:
class NewsItem(Item):
content = fields.TextField() # correct, no default value
author = fields.TextField(default=u'Myself") # correct, with default value
published = fields.DateField(default=23) # wrong default type (will raise TypeError)
System Message: ERROR/3 (<stdin>, line 207)
Unknown directive type "method".
.. method:: to_python(value)
Convert the input value to the type expected by this field and return
it.
For example, :class:`IntegerField` would convert ``'1'`` to ``1``, while
:class:`DecimalField` would convert ``'1'`` to ``Decimal('1')`` and so
on.
This method is not implemented in the :class:`BaseField` class, so it
must always be implemented in all its subclasses, in order to be usable.
This method should raise ``TypeError`` if the input type is not
supported, and ``ValueError`` if the input type is support but its value
is not appropriate (for example, an integer outside a given range).
This method must always return object of the expected field type.
System Message: ERROR/3 (<stdin>, line 225)
Unknown directive type "method".
.. method:: get_default() Return the default value for this field, or ``None`` if the field doesn't specify any.