diff --git a/scrapy/trunk/docs/topics/items.rst b/scrapy/trunk/docs/topics/items.rst index 3a6385379..6a915fa4d 100644 --- a/scrapy/trunk/docs/topics/items.rst +++ b/scrapy/trunk/docs/topics/items.rst @@ -1,117 +1,129 @@ .. _topics-items: -================ -Items & Adaptors -================ +===== +Items +===== + +.. module:: scrapy.item + :synopsis: Objects for storing scraped data + +Quick overview +============== | In Scrapy, items are the placeholder to use for the scraped data. - They are represented by a ScrapedItem object, or any descendant class instance, and store the information in class attributes. + They are represented by a :class:`ScrapedItem` object, or any descendant class instance, and store the information in class attributes. -These attributes are set by using the item's ``attribute`` method, for example:: +ScrapedItems +============ - person = ScrapedItem() - person.attribute('name', 'John') - person.attribute('age', 35) +.. class:: ScrapedItem + +Methods +------- + +.. method:: ScrapedItem.__init__(data={}) + + Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data`` dict. + +.. method:: ScrapedItem.attribute(self, attrname, value, override=False, add=False, **kwargs) + + Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the attribute's adaptor pipeline (if any). + + ``attrname`` is a string containing the name of the attribute you're setting. + + ``value`` is the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any). + + ``override``, if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what. + + ``add``, if True, tries to concatenate the given ``value`` with the one already set in the item. This will work as long as + the old value is a list (in which case the new value will be appended, or the list will be extended if both are lists), + or as long as both values are strings (in which case ``add`` will be used as the delimiter, or default to '' if ``add=True``). + + ``kwargs`` - any extra parameters will be passed to any adaptor that receives an 'adaptor_args' parameter as a dictionary. + Check the Adaptors reference for more information. + +.. method:: ScrapedItem.set_adaptors(self, adaptors_dict) + + Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline. + +.. method:: ScrapedItem.set_attrib_adaptors(self, attrib, pipe) + + Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``) + +.. method:: ScrapedItem.add_adaptor(self, attrib, adaptor, position=None) + + Adds an adaptor to an already existing (or not) pipeline. + + ``attr`` is the name of the attribute you're adding adaptors to. + + ``adaptor`` is a callable to be added to the pipeline. + + ``position`` is an integer representing the place where to add the adaptor. + If it's ``None``, the adaptor will be appended at the end of the pipeline. + +Examples +-------- + +Setting some basic attributes to a newly created item:: + + >>> from scrapy.item import ScrapedItem + >>> person = ScrapedItem() + >>> person.attribute('name', 'John') + >>> person.attribute('age', 35) + >>> person + ScrapedItem({'age': 35, 'name': 'John'}) + +We can also create an item and set its attributes by passing them inline using a dictionary, like:: + + >>> person = ScrapedItem({'name': 'John', 'age': 35}) + >>> person + ScrapedItem({'age': 35, 'name': 'John'}) + +Also, notice that making consecutive calls to the attribute method does *not* change its value, unless you use the `override` parameter:: + + >>> person = ScrapedItem() + >>> person.attribute('name', 'John') + >>> person + ScrapedItem({'name': 'John'}) + + >>> person.attribute('name', 'Charlie') + >>> person + ScrapedItem({'name': 'John'}) + + >>> person.attribute('name', 'Charlie', override=True) + >>> person + ScrapedItem({'name': 'Charlie'}) + +There's also an `add` parameter useful for concatenating lists or strings given a delimiter (or not):: + + >>> person = ScrapedItem() + >>> person.attribute('name', 'John') + >>> person + ScrapedItem({'name': 'John'}) + + # If add is True, '' is used as the default delimiter for joining strings + >>> person.attribute('name', 'Doe', add=True) + >>> person + ScrapedItem({'name': 'JohnDoe'}) + + # Otherwise, you can specify your own delimiter + >>> person.attribute('name', 'Smith', add=' ') + >>> person + ScrapedItem({'name': 'JohnDoe Smith'}) + + >>> person.attribute('children', ['Ken', 'Tom']) + >>> person + ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom']}) + + # You can also append to lists... + >>> person.attribute('children', 'Billy', add=True) + >>> person + ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy']}) + + # And even extend them + >>> person.attribute('children', ['Dan', 'George'], add=True) + >>> person + ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy', 'Dan', 'George']}) Now, normally when we're scraping an HTML file, or almost any kind of file, information doesn't come to us exactly as we need it. We usually have to make some adaptations here and there; and that's when the adaptors enter the game. - -Adaptors --------- - -| Adaptors are basically functions that receive one value (advanced adaptors may receive more, but we'll see that later), modify it, and return - a new value. -| In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes. -| Adaptor pipelines are nothing else but a list of adaptors which will be iterated, calling each adaptor and passing the values from one to each other. - -| The most common example use of adaptors appears when parsing HTML pages. To do this, we normally use XPathSelectors which need to be extracted some way. -| You could extract them yourself, as well as doing any kind of adaptation before assigning, but the idea of adaptor pipelines is to simplify this task, and the spider's code. - -So let's imagine that you want to scrape some information from a page as follows:: - - - - - - - - - - - -
Manufacturer/NameWeight/UnitPrice
- 1000 gr.$ 25
- -You can test yourself with this page, since it actually exists here -> [URL] -Open a Scrapy shell by doing:: - - ./scrapy-ctl.py shell [URL] - -And then let's try to create the item ourselves. Something like:: - - >> from scrapy.item import ScrapedItem - >> item = ScrapedItem() - >> item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()')) - >> item.manufacturer - << ['John & Bill's farm - Bananas'] - -| Okay, what we did here was creating an item, and setting its 'manufacturer' attribute by using the selector that Scrapy already created for us when the response was downloaded. -| As you can see, we didn't apply the extract method to the selector, but the data got extracted anyway. This is because Scrapy uses scrapy.contrib.adaptors.extract as the default - adaptor for every attribute, which tries to extract any selector given, or otherwise returns a list containing the received data. - -Anyway, that scraped data needs a bit more processing, what about this?:: - - >> item = ScrapedItem() - >> item.add_adaptor('manufacturer', adaptors.Unquote()) - >> item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()').re(r'^(.*?) -')) - >> item.manufacturer - << ['John & Bill's farm'] - -| Well, looks much cooler now :) - -Let's now try to make a spider to scrape this page:: - - from decimal import Decimal - from scrapy.item import ScrapedItem - from scrapy.contrib import adaptors - from scrapy.contrib.spiders import CrawlSpider, Rule - from scrapy.xpath.selector import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - - class MySpider(CrawlSpider): - domain_name = 'example.com' - start_urls = ['http://example.com/items'] - - rules = ( - Rule(RegexLinkExtractor(allow=(r'item\d+\.html', )), 'parse_item'), - ) - - def parse_item(self, response): - item = ScrapedItem() - item.add_adaptor('manufacturer', adaptors.Unquote()) - item.add_adaptor('price', adaptors.Delist()) - item.add_adaptor('price', Decimal) - - item.attribute('manufacturer', hxs.x('//td[@id="product_name"]/text()').re(r'^(.*?) -')) - item.attribute('name', hxs.x('//td[@id="product_name"]/text()').re(r'- (.*)$')) - item.attribute('weight', hxs.x('//td[@id="product_weight"]/text()')) - item.attribute('price', hxs.x('//td[@id="product_price"]/text()').re(r'$\s*(\d+)')) - - return [item] - - SPIDER = MySpider() - - -| Basically this spider looks for the product name in the page, splits it in two by using regular expressions and gets the manufacturer - and product name. -| The manufacturer name may contain entities, as we could see, so we added the ``Unquote`` adaptor to its pipeline. In a real life case, you should probably - add it to the name attribute too, but it doesn't matter here. -| In order to parse the price, we added two adaptors: Delist, an adaptor that takes care of joining the list returned by the extractor, and Decimal, a class - from Python's decimal module, whose constructor receives a string and returns a Decimal object. - -Scraping the sample page with this code would give us an item similar to:: - - ScrapedItem(name='Bananas', manufacturer='John & Bill's farm', weight='1000 gr.', price=Decimal('25')) - -There could be more parsing done here through adaptors, like parsing the weight according to its unit, and more; but i'll let you practice on your own. -