2.8 KiB
Items
System Message: ERROR/3 (<stdin>, line 9)
Unknown directive type "currentmodule".
.. currentmodule:: scrapy.item
The goal of the scraping process is to obtain scraped items from scraped pages.
Basic scraped items
In Scrapy the items are represented by a :class:`ScrapedItem` (almost an empty class) or any subclass of it.
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "class".To use :class:`ScrapedItem` you simply instantiate it and use instance attributes to store the information.
System Message: ERROR/3 (<stdin>, line 19); backlink
Unknown interpreted text role "class".
>>> from scrapy.item import ScrapedItem
>>> item = ScrapedItem()
>>> item.headline = 'Headline'
>>> item.content = 'Content'
>>> item.published = '2009-07-08'
>>> item
ScrapedItem({'headline': 'Headline', 'content': 'Content', 'published': '2009-07-08'})
Or you can use your own class to represent items, just be sure it inherits from :class:`ScrapedItem`.
System Message: ERROR/3 (<stdin>, line 30); backlink
Unknown interpreted text role "class".More advanced items
System Message: ERROR/3 (<stdin>, line 38)
Unknown directive type "currentmodule".
.. currentmodule:: scrapy.newitem
Scrapy provides :class:`Item` (a subclass of :class:`~scrapy.item.ScrapedItem`) that works like a form with fields to store the item's data.
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "class".To use this items you first define the item's fields as class attributes:
from scrapy.newitem import Item
from scrapy.newitem import fields
class NewsItem(Item):
headline = fields.TextField()
content = fields.TextField()
published = fields.DateField()
And then you instantiate the item and assign values to its fields, which will be converted to the expected Python types depending of their class:
>>> item = NewsItem() >>> item['headline'] = u'Headline' >>> item['content'] = u'Content' >>> item['published'] = '2009-07-08' >>> item NewsItem(headline=u'Headline', content=u'Content', published=datetime.date(2009, 7, 8))
Using this may seen complicated at first, but gives you much power over scraped data, like :ref:`topics-newitem-index-defaults`, :ref:`topics-newitem-adaptors`, etc.
System Message: ERROR/3 (<stdin>, line 63); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 63); backlink
Unknown interpreted text role "ref".Default values for fields
Each field accepts a default argument, that sets the default value of the field.
Fields which contain a default value will always return that value when not set, while fields which don't contain a default value will raise KeyError, you can use :meth:`Item.get` method to avoid this.
System Message: ERROR/3 (<stdin>, line 75); backlink
Unknown interpreted text role "meth".System Message: WARNING/2 (<stdin>, line 79)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.newitem import Item, fields
class NewsItem(Item):
headline = fields.TextField()
content = fields.TextField()
published = fields.DateField()
author = fields.TextField(default=u'Myself')
views = fields.IntegerField(default=0)
System Message: WARNING/2 (<stdin>, line 90)
Cannot analyze code. Pygments package not found.
.. code-block:: python
>>> item = NewsItem()
>>> item['author']
u'Myself'
>>> item['views']
0
>>> item['headline']
Traceback (most recent call last):
...
KeyError: 'headline
>>> item.get('headline') is None
True