mirror of https://github.com/scrapy/scrapy.git
Updated items documentation
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40649
This commit is contained in:
elpolilla
parent
f1962c1c0a
commit
111ba5d583
|
|
@ -1,117 +1,129 @@
|
|||
.. _topics-items:
|
||||
|
||||
================
|
||||
Items & Adaptors
|
||||
================
|
||||
=====
|
||||
Items
|
||||
=====
|
||||
|
||||
.. module:: scrapy.item
|
||||
:synopsis: Objects for storing scraped data
|
||||
|
||||
Quick overview
|
||||
==============
|
||||
|
||||
| In Scrapy, items are the placeholder to use for the scraped data.
|
||||
They are represented by a ScrapedItem object, or any descendant class instance, and store the information in class attributes.
|
||||
They are represented by a :class:`ScrapedItem` object, or any descendant class instance, and store the information in class attributes.
|
||||
|
||||
These attributes are set by using the item's ``attribute`` method, for example::
|
||||
ScrapedItems
|
||||
============
|
||||
|
||||
person = ScrapedItem()
|
||||
person.attribute('name', 'John')
|
||||
person.attribute('age', 35)
|
||||
.. class:: ScrapedItem
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: ScrapedItem.__init__(data={})
|
||||
|
||||
Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data`` dict.
|
||||
|
||||
.. method:: ScrapedItem.attribute(self, attrname, value, override=False, add=False, **kwargs)
|
||||
|
||||
Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the attribute's adaptor pipeline (if any).
|
||||
|
||||
``attrname`` is a string containing the name of the attribute you're setting.
|
||||
|
||||
``value`` is the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any).
|
||||
|
||||
``override``, if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what.
|
||||
|
||||
``add``, if True, tries to concatenate the given ``value`` with the one already set in the item. This will work as long as
|
||||
the old value is a list (in which case the new value will be appended, or the list will be extended if both are lists),
|
||||
or as long as both values are strings (in which case ``add`` will be used as the delimiter, or default to '' if ``add=True``).
|
||||
|
||||
``kwargs`` - any extra parameters will be passed to any adaptor that receives an 'adaptor_args' parameter as a dictionary.
|
||||
Check the Adaptors reference for more information.
|
||||
|
||||
.. method:: ScrapedItem.set_adaptors(self, adaptors_dict)
|
||||
|
||||
Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline.
|
||||
|
||||
.. method:: ScrapedItem.set_attrib_adaptors(self, attrib, pipe)
|
||||
|
||||
Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``)
|
||||
|
||||
.. method:: ScrapedItem.add_adaptor(self, attrib, adaptor, position=None)
|
||||
|
||||
Adds an adaptor to an already existing (or not) pipeline.
|
||||
|
||||
``attr`` is the name of the attribute you're adding adaptors to.
|
||||
|
||||
``adaptor`` is a callable to be added to the pipeline.
|
||||
|
||||
``position`` is an integer representing the place where to add the adaptor.
|
||||
If it's ``None``, the adaptor will be appended at the end of the pipeline.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Setting some basic attributes to a newly created item::
|
||||
|
||||
>>> from scrapy.item import ScrapedItem
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person.attribute('age', 35)
|
||||
>>> person
|
||||
ScrapedItem({'age': 35, 'name': 'John'})
|
||||
|
||||
We can also create an item and set its attributes by passing them inline using a dictionary, like::
|
||||
|
||||
>>> person = ScrapedItem({'name': 'John', 'age': 35})
|
||||
>>> person
|
||||
ScrapedItem({'age': 35, 'name': 'John'})
|
||||
|
||||
Also, notice that making consecutive calls to the attribute method does *not* change its value, unless you use the `override` parameter::
|
||||
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
|
||||
>>> person.attribute('name', 'Charlie')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
|
||||
>>> person.attribute('name', 'Charlie', override=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'Charlie'})
|
||||
|
||||
There's also an `add` parameter useful for concatenating lists or strings given a delimiter (or not)::
|
||||
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
|
||||
# If add is True, '' is used as the default delimiter for joining strings
|
||||
>>> person.attribute('name', 'Doe', add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe'})
|
||||
|
||||
# Otherwise, you can specify your own delimiter
|
||||
>>> person.attribute('name', 'Smith', add=' ')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith'})
|
||||
|
||||
>>> person.attribute('children', ['Ken', 'Tom'])
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom']})
|
||||
|
||||
# You can also append to lists...
|
||||
>>> person.attribute('children', 'Billy', add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy']})
|
||||
|
||||
# And even extend them
|
||||
>>> person.attribute('children', ['Dan', 'George'], add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy', 'Dan', 'George']})
|
||||
|
||||
Now, normally when we're scraping an HTML file, or almost any kind of file, information doesn't come to us exactly as we need it. We usually
|
||||
have to make some adaptations here and there; and that's when the adaptors enter the game.
|
||||
|
||||
Adaptors
|
||||
--------
|
||||
|
||||
| Adaptors are basically functions that receive one value (advanced adaptors may receive more, but we'll see that later), modify it, and return
|
||||
a new value.
|
||||
| In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes.
|
||||
| Adaptor pipelines are nothing else but a list of adaptors which will be iterated, calling each adaptor and passing the values from one to each other.
|
||||
|
||||
| The most common example use of adaptors appears when parsing HTML pages. To do this, we normally use XPathSelectors which need to be extracted some way.
|
||||
| You could extract them yourself, as well as doing any kind of adaptation before assigning, but the idea of adaptor pipelines is to simplify this task, and the spider's code.
|
||||
|
||||
So let's imagine that you want to scrape some information from a page as follows::
|
||||
|
||||
<table id='products_info'>
|
||||
<tr>
|
||||
<td>Manufacturer/Name</td>
|
||||
<td>Weight/Unit</td>
|
||||
<td>Price</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td id='product_name>John & Bill's farm - Bananas</td>
|
||||
<td id='product_weight'>1000 gr.</td>
|
||||
<td id='product_price'>$ 25</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
You can test yourself with this page, since it actually exists here -> [URL]
|
||||
Open a Scrapy shell by doing::
|
||||
|
||||
./scrapy-ctl.py shell [URL]
|
||||
|
||||
And then let's try to create the item ourselves. Something like::
|
||||
|
||||
>> from scrapy.item import ScrapedItem
|
||||
>> item = ScrapedItem()
|
||||
>> item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()'))
|
||||
>> item.manufacturer
|
||||
<< ['John & Bill's farm - Bananas']
|
||||
|
||||
| Okay, what we did here was creating an item, and setting its 'manufacturer' attribute by using the selector that Scrapy already created for us when the response was downloaded.
|
||||
| As you can see, we didn't apply the extract method to the selector, but the data got extracted anyway. This is because Scrapy uses scrapy.contrib.adaptors.extract as the default
|
||||
adaptor for every attribute, which tries to extract any selector given, or otherwise returns a list containing the received data.
|
||||
|
||||
Anyway, that scraped data needs a bit more processing, what about this?::
|
||||
|
||||
>> item = ScrapedItem()
|
||||
>> item.add_adaptor('manufacturer', adaptors.Unquote())
|
||||
>> item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()').re(r'^(.*?) -'))
|
||||
>> item.manufacturer
|
||||
<< ['John & Bill's farm']
|
||||
|
||||
| Well, looks much cooler now :)
|
||||
|
||||
Let's now try to make a spider to scrape this page::
|
||||
|
||||
from decimal import Decimal
|
||||
from scrapy.item import ScrapedItem
|
||||
from scrapy.contrib import adaptors
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
|
||||
class MySpider(CrawlSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://example.com/items']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=(r'item\d+\.html', )), 'parse_item'),
|
||||
)
|
||||
|
||||
def parse_item(self, response):
|
||||
item = ScrapedItem()
|
||||
item.add_adaptor('manufacturer', adaptors.Unquote())
|
||||
item.add_adaptor('price', adaptors.Delist())
|
||||
item.add_adaptor('price', Decimal)
|
||||
|
||||
item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()').re(r'^(.*?) -'))
|
||||
item.attribute('name', hxs.x('//td[@id="product_name"]/text()').re(r'- (.*)$'))
|
||||
item.attribute('weight', hxs.x('//td[@id="product_weight"]/text()'))
|
||||
item.attribute('price', hxs.x('//td[@id="product_price"]/text()').re(r'$\s*(\d+)'))
|
||||
|
||||
return [item]
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
|
||||
| Basically this spider looks for the product name in the page, splits it in two by using regular expressions and gets the manufacturer
|
||||
and product name.
|
||||
| The manufacturer name may contain entities, as we could see, so we added the ``Unquote`` adaptor to its pipeline. In a real life case, you should probably
|
||||
add it to the name attribute too, but it doesn't matter here.
|
||||
| In order to parse the price, we added two adaptors: Delist, an adaptor that takes care of joining the list returned by the extractor, and Decimal, a class
|
||||
from Python's decimal module, whose constructor receives a string and returns a Decimal object.
|
||||
|
||||
Scraping the sample page with this code would give us an item similar to::
|
||||
|
||||
ScrapedItem(name='Bananas', manufacturer='John & Bill's farm', weight='1000 gr.', price=Decimal('25'))
|
||||
|
||||
There could be more parsing done here through adaptors, like parsing the weight according to its unit, and more; but i'll let you practice on your own.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue