mirror of https://github.com/scrapy/scrapy.git
Updated adaptors and items documentation
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40794
This commit is contained in:
elpolilla
parent
ff88171b05
commit
9634af9aa7
|
|
@ -12,307 +12,135 @@ Adaptors
|
|||
Quick overview
|
||||
==============
|
||||
|
||||
Adaptors are basically functions that receive one value (advanced adaptors may
|
||||
receive more, but we'll see that later), modify it, and return a new value.
|
||||
Scrapy's adaptors are a nice feature attached to RobustScrapedItems that allow you
|
||||
to easily modify (adapt to your needs) any kind of information you want to put in your items at assignation time.
|
||||
|
||||
In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes.
|
||||
The following diagram shows the data flow from the moment you call the `attribute` method until the attribute is
|
||||
actually set.
|
||||
|
||||
Adaptor pipelines are nothing else but a list of adaptors which will be
|
||||
iterated at the moment of assigning an attribute, calling each adaptor and
|
||||
passing the values from one to each other.
|
||||
.. image:: _images/adaptors_diagram.png
|
||||
|
||||
Example use
|
||||
===========
|
||||
As you can see, adaptor pipelines are executed in tree form; which means that, for each of the values you pass to
|
||||
the `attribute` method, the first adaptor will be applied. Then, for each of the resulting values of the first adaptor,
|
||||
the second adaptor will be called, and so on.
|
||||
This process will end up with a list of adapted values, which may contain zero, one, or many values.
|
||||
|
||||
The most common example use of adaptors appears when parsing HTML pages. To do
|
||||
this, we normally use XPathSelectors to retrieve data, which need to be
|
||||
extracted some way, and in most cases, filtered some way.
|
||||
In case the attribute is a single-valued (this is defined in the item's ATTRIBUTES dictionary), the first element of this
|
||||
list will be set, unless you call the `attribute` method with the add parameter as True, in which case the item's method
|
||||
`_add_single_attributes` will be called with the attribute's name, type, and the list of attributes to join as parameters.
|
||||
By default, this method raises NotImplementedError, so you should override it in your items in order to join any kind of objects.
|
||||
|
||||
You could extract them yourself, as well as doing any kind of adaptation before
|
||||
assigning, but the idea of adaptor pipelines is to simplify this task, and the
|
||||
spider's code.
|
||||
If the attribute is a multivalued, the resulting list will be set to the item as is, unless you use -again- add=True,
|
||||
in which case the list of already-existing values (if any) will be extended with the new one.
|
||||
|
||||
Let's imagine, for example, that you want to scrape information from pages as
|
||||
the following:
|
||||
Adaptor Pipelines
|
||||
=================
|
||||
|
||||
.. literalinclude:: ../_static/items_adaptors-sample1.html
|
||||
:language: html
|
||||
.. class:: AdaptorPipe(adaptors=None)
|
||||
|
||||
In this case, you'd have to scrape some products information, like their
|
||||
manufacturer, name, and price. We'll put this information inside ScrapedItems
|
||||
and attach them some adaptors to process our data better. You can test yourself
|
||||
with this page, since it actually exists `here
|
||||
<../_static/items_adaptors-sample1.html>`_
|
||||
An instance of this class represents an adaptor pipeline to be set for adapting a certain
|
||||
item's attribute.
|
||||
It provides some useful methods for adding/removing adaptors, and takes care of executing them properly.
|
||||
Usually this class is not used directly, since the items already provide ways to handle adaptors without
|
||||
having to manage AdaptorPipes.
|
||||
|
||||
.. highlight:: sh
|
||||
:param adaptors: A list of callables to be added as adaptors at instancing time.
|
||||
|
||||
Open it in a Scrapy shell by doing::
|
||||
Methods:
|
||||
|
||||
./scrapy-ctl.py shell 'http://doc.scrapy.org/_static/items_adaptors-sample1.html'
|
||||
.. method:: add_adaptor(adaptor, position=None)
|
||||
|
||||
.. highlight:: python
|
||||
This method is used for adding adaptors to the pipeline given a certain position.
|
||||
|
||||
And then let's try to find the products and scrape an item manually. Something like::
|
||||
:param adaptor: Any callable that works as an adaptor
|
||||
:param position: An integer meaning the position in which the adaptor will be inserted. If it's None
|
||||
the adaptor will be appended at the end of the pipeline.
|
||||
|
||||
>>> product_rows = hxs.x('//tr[child::td[@class="prod_attrib"]]')
|
||||
>>> product_rows
|
||||
[<HtmlXPathSelector (tr) xpath=//tr[child::td[@class="prod_attrib"]]>,
|
||||
<HtmlXPathSelector (tr) xpath=//tr[child::td[@class="prod_attrib"]]>]
|
||||
Usage
|
||||
=====
|
||||
|
||||
# Now we'll play trying to see how we scrape the first product
|
||||
>>> from scrapy.item import ScrapedItem
|
||||
>>> item = ScrapedItem()
|
||||
>>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()'))
|
||||
>>> item.manufacturer
|
||||
[<HtmlXPathSelector (text) xpath=td[@class="prod_attrib"][1]/text()>]
|
||||
As it was previously said, in order to use adaptor pipelines you must inherit your items from the RobustScrapedItem class.
|
||||
If you don't know anything about these items, read the :ref:`topics-items` reference first.
|
||||
|
||||
Okay, what we did here was creating an item, and setting its 'manufacturer'
|
||||
attribute by using the selectors we already had for each product row.
|
||||
Once you've created your own item class (inherited from RobustScrapedItem) with the attributes you're going to use,
|
||||
you have to add adaptor pipelines to each attribute you'd like to adapt data for.
|
||||
For doing so, RobustScrapedItems provide some useful methods (`set_adaptors`, `set_attrib_adaptors`, and more), which are
|
||||
also described in its reference.
|
||||
|
||||
As you can see, we didn't apply the extract method to the selector, so the data
|
||||
stored in the product was exactly the same that our selector's call to the
|
||||
``x`` method returns; another selector.
|
||||
|
||||
We could extract the information before setting it, but we'll use adaptors instead:
|
||||
|
||||
>>> from scrapy.contrib_exp import adaptors
|
||||
>>> item = ScrapedItem()
|
||||
>>> item.add_adaptor('manufacturer', adaptors.extract)
|
||||
>>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()'))
|
||||
>>> item.manufacturer
|
||||
[u"Bill & Ted's Farm"]
|
||||
|
||||
Better now, right? At least we have some readable data :)
|
||||
|
||||
Although we'd probably want to remove those entities, and to store the
|
||||
information as a string...
|
||||
|
||||
>>> item = ScrapedItem()
|
||||
>>> item.set_attrib_adaptors('manufacturer', [
|
||||
adaptors.extract,
|
||||
adaptors.Unquote(),
|
||||
adaptors.Delist() ])
|
||||
>>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()'))
|
||||
>>> item.manufacturer
|
||||
u"Bill & Ted's Farm"
|
||||
|
||||
Cool, now that looks like something that can be stored correctly. However, we
|
||||
must look at the rest of the attributes too. At first sight, it looks like at
|
||||
least the name and the description could use the same adaptors as the
|
||||
manufacturer, since they're all a simple text.
|
||||
|
||||
The weight and the price could get a better parsing though; at least to convert
|
||||
them to real decimals. Let's experiment a bit more::
|
||||
|
||||
>>> from decimal import Decimal # We'll use Decimal objects for storing the price and weight
|
||||
|
||||
# We'll also use the same item, since in this case we'll be working with a different attribute
|
||||
>>> item.set_attrib_adaptors('price', [ adaptors.Delist(), Decimal ])
|
||||
>>> item.attribute('price', product_rows[0].x('td[@class="prod_attrib"][5]/text()').re('(\d+)'))
|
||||
>>> item.price
|
||||
Decimal('300')
|
||||
|
||||
In this case it wasn't necessary to use the extract adaptor, because applying
|
||||
the ``re`` method over a selector already extracts the resulting content.
|
||||
|
||||
Now, the price was quite easy, but the weight is a bit tricky, for the simple
|
||||
fact that it has more than one weighing unit. In this case, we'll have to write
|
||||
our own little adaptor that takes care of parsing a string and returning a
|
||||
Decimal according to its unit.
|
||||
|
||||
Something like this could work, although it's very primitive::
|
||||
|
||||
def parse_weight(string):
|
||||
'''This adaptor receives a string and tries to parse it
|
||||
as weight, converting it to grams and returning a Decimal object'''
|
||||
|
||||
conversions = {
|
||||
'kg': Decimal('0.001'),
|
||||
'gr': Decimal('1'),
|
||||
'mg': Decimal('1000'),
|
||||
}
|
||||
|
||||
quantity = re.search('(\d+)', string)
|
||||
if not quantity:
|
||||
return Decimal('0')
|
||||
quantity = Decimal(quantity.group(1))
|
||||
|
||||
for unit in conversions:
|
||||
if unit in string:
|
||||
quantity = quantity / conversions[unit]
|
||||
break
|
||||
return quantity
|
||||
|
||||
And now some testing::
|
||||
|
||||
>>> parse_weight('4kg')
|
||||
Decimal('4000')
|
||||
>>> parse_weight('200 gr.')
|
||||
Decimal('200')
|
||||
>>> parse_weight('100 mg')
|
||||
Decimal('0.1')
|
||||
|
||||
>>> item.set_attrib_adaptors('weight', [
|
||||
adaptors.extract,
|
||||
adaptors.Delist(),
|
||||
parse_weight ])
|
||||
>>> item.attribute('weight', product_rows[0].x('td[@class="prod_attrib"][4]/text()').re('(\d+)'))
|
||||
>>> item.weight
|
||||
Decimal('2000')
|
||||
|
||||
Ok, done! Let's now sum this up into a spider::
|
||||
|
||||
from decimal import Decimal
|
||||
from scrapy.item import ScrapedItem
|
||||
from scrapy.contrib_exp import adaptors
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
|
||||
def parse_weight(string):
|
||||
conversions = {
|
||||
'kg': Decimal('0.001'),
|
||||
'gr': Decimal('1'),
|
||||
'mg': Decimal('1000'),
|
||||
}
|
||||
|
||||
quantity = re.search('(\d+)', string)
|
||||
if not quantity:
|
||||
return Decimal('0')
|
||||
quantity = Decimal(quantity.group(1))
|
||||
|
||||
for unit in conversions:
|
||||
if unit in string:
|
||||
quantity = quantity / conversions[unit]
|
||||
break
|
||||
return quantity
|
||||
|
||||
class MySpider(CrawlSpider):
|
||||
domain_name = 'scrapy.org'
|
||||
start_urls = ['http://doc.scrapy.org/_static/items_adaptors-sample1.html']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=(r'sample\d+\.html', )), 'parse_page'),
|
||||
)
|
||||
|
||||
adaptors = {
|
||||
'manufacturer': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ],
|
||||
'name': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ],
|
||||
'description': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ],
|
||||
'weight': [ adaptors.extract, adaptors.Delist(), parse_weight ],
|
||||
'price': [ adaptors.Delist(), Decimal ]
|
||||
}
|
||||
|
||||
def parse_page(self, response):
|
||||
items = []
|
||||
rows = hxs.x('//tr[child::td[@class="prod_attrib"]]')
|
||||
for product in rows:
|
||||
item = ScrapedItem()
|
||||
item.set_adaptors(self.adaptors)
|
||||
|
||||
item.attribute('manufacturer', product.x('td[@class="prod_attrib"][1]/text()'))
|
||||
item.attribute('name', product.x('td[@class="prod_attrib"][2]/text()'))
|
||||
item.attribute('description', product.x('td[@class="prod_attrib"][3]/text()'))
|
||||
item.attribute('weight', product.x('td[@class="prod_attrib"][4]/text()'))
|
||||
item.attribute('price', product.x('td[@class="prod_attrib"][5]/text()').re('(\d+)'))
|
||||
items.append(item)
|
||||
|
||||
return items
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
|
||||
Basically this spider looks for the product rows in the page, creates an item
|
||||
for each of them, attaches them some adaptors, and fills their attributes in.
|
||||
|
||||
Scraping the sample page with this code would give us these items::
|
||||
|
||||
ScrapedItem({u"name": u"Bananas", u"manufacturer": u"Bill & Ted's farm", u"description": "Delicious fruit", u"weight": Decimal("2000"), u"price": Decimal("300")})
|
||||
ScrapedItem({u"name": u"Apple pie", u"manufacturer": u"Grandma's", u"description": "Grandma's best dish", u"weight": Decimal("250"), u"price": Decimal("200")})
|
||||
|
||||
There could be more parsing done here through adaptors, like parsing different
|
||||
price currencies, or more advanced weight parsing (this one was very, very,
|
||||
simple and buggy). Nevertheless, I hope that this was useful as an example use
|
||||
of adaptors.
|
||||
|
||||
|
||||
More complex adaptors
|
||||
=====================
|
||||
|
||||
Okay, what happens when you have adaptors that receive extra parameters in
|
||||
order to modify their behaviour? For example, going back to money, what if you
|
||||
have a function capable of converting from one currency to another one by
|
||||
receiving three parameters: the value, its currency, and the currency to
|
||||
convert to?. Well, don't worry, Scrapy can handle this situation and use this
|
||||
adaptor too, and in fact, its something quite easy to do.
|
||||
|
||||
Basically the difference between a regular adaptor and a "complex" one (I call
|
||||
them complex just for the fact that the others are *very* simple), is that the
|
||||
first one always receives and returns a single value, while the others may
|
||||
receive more, passed by you to the item's ``attribute`` method. Whenever this
|
||||
method (``attribute``) receives any extra parameters (apart from the value to
|
||||
set, ``override``, and ``add``), it automatically passes them to any adaptor
|
||||
that receives a parameter called ``adaptor_args`` in a dictionary. This
|
||||
adaptor should read the dictionary and check if it has received any parameters
|
||||
there.
|
||||
|
||||
Notice that you should be careful with the naming of these parameters in order
|
||||
to not generate confusion between adaptors, because every adaptor (that has the
|
||||
``adaptor_args`` parameter, of course) will receive the same dict of keywords,
|
||||
containing keywords that are destined for itself, and some that aren't.
|
||||
|
||||
For example, it's not very wise to use a parameter called 'encoding' unless
|
||||
you'd like all the adaptors to receive it; otherwise it'd be better to use the
|
||||
adaptor's name as a prefix to the parameter, just to make it clearer.
|
||||
|
||||
Let's now see the currency example, now written in code.
|
||||
|
||||
First, we'll define a class called Currency and a dictionary, in order to store
|
||||
some currencies and their exchange rate::
|
||||
|
||||
class Currency(object):
|
||||
def __init__(self, from_dollars, to_dollars):
|
||||
self._from_dollars = from_dollars
|
||||
self._to_dollars = to_dollars
|
||||
|
||||
def is_decimal(self, value):
|
||||
if not isinstance(value, Decimal):
|
||||
raise Exception('Value must be a Decimal object')
|
||||
return True
|
||||
|
||||
def from_dollars(self, value):
|
||||
if self.is_decimal(value):
|
||||
return value * self._from_dollars
|
||||
|
||||
def to_dollars(self, value):
|
||||
if self.is_decimal(value):
|
||||
return value * self._to_dollars
|
||||
|
||||
currencies = {
|
||||
'dollar': Currency(from_dollars=Decimal('1'), to_dollars=Decimal('1')),
|
||||
'euro': Currency(from_dollars=Decimal('0.7294'), to_dollars=Decimal('1.3710')),
|
||||
'yen': Currency(from_dollars=Decimal('90.82'), to_dollars=Decimal('0.0110')),
|
||||
'pound': Currency(from_dollars=Decimal('0.6547'), to_dollars=Decimal('1.5274')),
|
||||
}
|
||||
|
||||
Ok, now that we've got some exchange information, the only thing missing is the
|
||||
adaptor that makes use of it::
|
||||
|
||||
def exchange_adaptor(value, adaptor_args):
|
||||
currency_from = adaptor_args.get('currency_from', 'dollar')
|
||||
currency_to = adaptor_args.get('currency_to', 'dollar')
|
||||
But let's now talk a bit about adaptors (singularly), what are them, and how should they be implemented?
|
||||
Adaptors are basically, any callable that receives a value, modifies it, and returns a new value (or more) so that the next
|
||||
adaptor goes on with another adapting task (or not).
|
||||
This is done this way to make the process of modifying information very customizable, and also to make adaptors reusable,
|
||||
since they are intended to be small functions designed for simple purposes that can be applied in many different cases.
|
||||
For example, you could make an adaptor for removing any <b> tags in a text, like this::
|
||||
|
||||
if currency_from in currencies and currency_to in currencies:
|
||||
dollars = currencies[currency_from].to_dollars(value)
|
||||
return currencies[currency_to].from_dollars(dollars)
|
||||
else:
|
||||
raise Exception('Unsupported currencies were specified')
|
||||
>>> B_TAG_RE = re.compile(r'</?b\s*>')
|
||||
>>> def remove_b_tags(text):
|
||||
>>> return B_TAG_RE.sub('', text)
|
||||
|
||||
And finally, testing it! ::
|
||||
Then you could easily add this adaptor to a certain attribute's pipeline like this::
|
||||
|
||||
>>> item = ScrapedItem()
|
||||
>>> item.set_attrib_adaptors('price', [ Decimal, exchange_adaptor ])
|
||||
>>> item.attribute('price', '20.5', currency_from='euro', currency_to='dollar')
|
||||
>>> item.price
|
||||
Decimal('28.10550')
|
||||
>>> item = MyItem()
|
||||
>>> item.add_adaptor('text', remove_b_tags)
|
||||
>>> item.attribute('text', u'<b>some random text in bold</b> and some random text in normal font')
|
||||
>>> item.text
|
||||
u'some random text in bold and some random text in normal font'
|
||||
|
||||
As you can see, this would make any value that you set to the item through the `attribute` method first pass through the
|
||||
`remove_b_tags` adaptor, which would also replace any matching tag with an empty string.
|
||||
|
||||
----
|
||||
|
||||
But anyway, let's now think of a bit more complicated (and useless) example: let's say you want to scrape a text, split it into single
|
||||
letters, strip the vowels, turn the rest to capital letters, and join them again.
|
||||
In this case, we could use three simple adaptors to process our data, plus a customized RobustScrapedItem for joining single
|
||||
text attributes; let's see an example::
|
||||
|
||||
>>> # First of all, we define the item class we're going to use
|
||||
>>> from string import ascii_letters
|
||||
>>> from scrapy.contrib.item import RobustScrapedItem
|
||||
>>> class MyItem(RobustScrapedItem):
|
||||
>>> ATTRIBUTES = {
|
||||
>>> 'text': basestring,
|
||||
>>> }
|
||||
|
||||
>>> def _add_single_attributes(self, attrname, attrtype, attributes):
|
||||
>>> return ''.join(attributes)
|
||||
|
||||
>>> # Now we'll write the needed adaptors
|
||||
>>> def to_letters(text):
|
||||
>>> return tuple(letter for letter in text)
|
||||
|
||||
>>> def is_vowel(letter):
|
||||
>>> if letter in ascii_letters and letter.lower() not in ('a', 'e', 'i', 'o', 'u'):
|
||||
>>> return letter
|
||||
|
||||
>>> def to_upper(letter):
|
||||
>>> return letter.upper()
|
||||
|
||||
>>> # Finally, we'll join all the pieces and see how it works
|
||||
>>> item = MyItem()
|
||||
>>> item.set_attrib_adaptors('text', [
|
||||
>>> to_letters,
|
||||
>>> is_vowel,
|
||||
>>> to_upper,
|
||||
>>> ])
|
||||
|
||||
Let's now try with an example text to see what happens::
|
||||
|
||||
>>> item.attribute('text', 'pi', 'wind', add=True)
|
||||
>>> item.text
|
||||
'PWND'
|
||||
|
||||
Debugging
|
||||
=========
|
||||
|
||||
While you're coding spiders and adaptors, you usually need to know exactly what does Scrapy
|
||||
do under the hood with the values you provide.
|
||||
There's a setting called :setting:`ADAPTORS_DEBUG` for this purpose that makes Scrapy print
|
||||
debugging messages each time an adaptors pipeline is run, specifying which attribute is being
|
||||
adapted data for, the input/output values of each adaptor in the pipeline, and the input/output
|
||||
of `_add_single_attributes` (in some cases).
|
||||
|
||||
You can enable this setting as any other, either by adding it to your settings file, or by enabling
|
||||
the environment variable `SCRAPY_ADAPTORS_DEBUG`.
|
||||
|
|
|
|||
|
|
@ -4,9 +4,6 @@
|
|||
Items
|
||||
=====
|
||||
|
||||
.. module:: scrapy.item
|
||||
:synopsis: Objects for storing scraped data
|
||||
|
||||
Quick overview
|
||||
==============
|
||||
|
||||
|
|
@ -16,114 +13,130 @@ Quick overview
|
|||
ScrapedItems
|
||||
============
|
||||
|
||||
.. module:: scrapy.item
|
||||
:synopsis: Objects for storing scraped data
|
||||
|
||||
.. class:: ScrapedItem
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: ScrapedItem.__init__(data={})
|
||||
.. method:: ScrapedItem.__init__(data=None)
|
||||
|
||||
Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data`` dict.
|
||||
:param data: A dictionary containing attributes and values to be set after instancing the item.
|
||||
|
||||
.. method:: ScrapedItem.attribute(self, attrname, value, override=False, add=False, **kwargs)
|
||||
|
||||
Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the attribute's adaptor pipeline (if any).
|
||||
|
||||
``attrname`` is a string containing the name of the attribute you're setting.
|
||||
|
||||
``value`` is the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any).
|
||||
|
||||
``override``, if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what.
|
||||
|
||||
``add``, if True, tries to concatenate the given ``value`` with the one already set in the item. This will work as long as
|
||||
the old value is a list (in which case the new value will be appended, or the list will be extended if both are lists),
|
||||
or as long as both values are strings (in which case ``add`` will be used as the delimiter, or default to '' if ``add=True``).
|
||||
|
||||
``kwargs`` - any extra parameters will be passed to any adaptor that receives an 'adaptor_args' parameter as a dictionary.
|
||||
Check the Adaptors reference for more information.
|
||||
|
||||
.. method:: ScrapedItem.set_adaptors(self, adaptors_dict)
|
||||
|
||||
Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline.
|
||||
|
||||
.. method:: ScrapedItem.set_attrib_adaptors(self, attrib, pipe)
|
||||
|
||||
Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``)
|
||||
|
||||
.. method:: ScrapedItem.add_adaptor(self, attrib, adaptor, position=None)
|
||||
|
||||
Adds an adaptor to an already existing (or not) pipeline.
|
||||
|
||||
``attr`` is the name of the attribute you're adding adaptors to.
|
||||
|
||||
``adaptor`` is a callable to be added to the pipeline.
|
||||
|
||||
``position`` is an integer representing the place where to add the adaptor.
|
||||
If it's ``None``, the adaptor will be appended at the end of the pipeline.
|
||||
Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data``
|
||||
dict (if any).
|
||||
These items are the most basic items available, and the common interface from which any items should inherit.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Setting some basic attributes to a newly created item::
|
||||
Creating an item and setting some attributes::
|
||||
|
||||
>>> from scrapy.item import ScrapedItem
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person.attribute('age', 35)
|
||||
>>> item = ScrapedItem()
|
||||
>>> item.name = 'John'
|
||||
>>> item.last_name = 'Smith'
|
||||
>>> item.age = 23
|
||||
>>> item
|
||||
ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'})
|
||||
|
||||
Creating an item and setting its attributes inline::
|
||||
|
||||
>>> person = ScrapedItem({'name': 'John', 'age': 23, 'last_name': 'Smith'})
|
||||
>>> person
|
||||
ScrapedItem({'age': 35, 'name': 'John'})
|
||||
ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'})
|
||||
|
||||
We can also create an item and set its attributes by passing them inline using a dictionary, like::
|
||||
RobustScrapedItems
|
||||
==================
|
||||
|
||||
>>> person = ScrapedItem({'name': 'John', 'age': 35})
|
||||
>>> person
|
||||
ScrapedItem({'age': 35, 'name': 'John'})
|
||||
.. module:: scrapy.contrib.item
|
||||
:synopsis: Objects for storing scraped data
|
||||
|
||||
Also, notice that making consecutive calls to the attribute method does *not* change its value, unless you use the `override` parameter::
|
||||
.. class:: RobustScrapedItem
|
||||
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
RobustScrapedItems are more complex items (compared to ScrapedItems) and have a few more features available, which
|
||||
include:
|
||||
|
||||
>>> person.attribute('name', 'Charlie')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
* Attributes dictionary: items that inherit from RobustScrapedItem are defined with a dictionary of attributes in the class.
|
||||
This allows the item to have much more logic at the moment of handling and setting attributes. The next features are
|
||||
built on top of this one.
|
||||
|
||||
>>> person.attribute('name', 'Charlie', override=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'Charlie'})
|
||||
* Adaptors: maybe the most important of the features that these items provide. The adaptors are a system designed for
|
||||
filtering/modifying data before setting it to the item, that makes cleansing tasks *a lot* easier.
|
||||
|
||||
There's also an `add` parameter useful for concatenating lists or strings given a delimiter (or not)::
|
||||
* Type checking: RobustScrapedItems come with a built-in type checking which assures you that no data of the wrong type will
|
||||
get into the items without raising a warning.
|
||||
|
||||
>>> person = ScrapedItem()
|
||||
>>> person.attribute('name', 'John')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'John'})
|
||||
* Versioning: These items also provide versioning by making a unique hash for each item based on its attributes values.
|
||||
|
||||
# If add is True, '' is used as the default delimiter for joining strings
|
||||
>>> person.attribute('name', 'Doe', add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe'})
|
||||
* ItemDeltas: You can subtract two RobustScrapedItems, which allows you to know the difference between a pair of items.
|
||||
This difference is represented by a RobustItemDelta object.
|
||||
|
||||
# Otherwise, you can specify your own delimiter
|
||||
>>> person.attribute('name', 'Smith', add=' ')
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith'})
|
||||
Methods
|
||||
-------
|
||||
|
||||
>>> person.attribute('children', ['Ken', 'Tom'])
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom']})
|
||||
.. method:: RobustScrapedItem.__init__(data=None, adaptor_args=None)
|
||||
|
||||
# You can also append to lists...
|
||||
>>> person.attribute('children', 'Billy', add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy']})
|
||||
:param data: Idem as in ScrapedItems
|
||||
:param adaptor_args: A dictionary of the like "attribute -> list of adaptors" for defining adaptors automatically after
|
||||
instancing the item.
|
||||
|
||||
# And even extend them
|
||||
>>> person.attribute('children', ['Dan', 'George'], add=True)
|
||||
>>> person
|
||||
ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy', 'Dan', 'George']})
|
||||
Constructor of RobustScrapedItem objects.
|
||||
|
||||
.. method:: RobustScrapedItem.attribute(self, attrname, value, override=False, add=False, ***kwargs)
|
||||
|
||||
Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the given attribute's adaptor
|
||||
pipeline (if any).
|
||||
|
||||
:param attrname: a string containing the name of the attribute you want to set.
|
||||
|
||||
:param value: the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any).
|
||||
|
||||
:param override: if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what.
|
||||
|
||||
:param add: if True, tries to concatenate the given ``value`` with the one already set in the item.
|
||||
For multivalued attributes, this will extend the list of already-set values, with the new ones.
|
||||
For single valued attributes, the method _add_single_attributes (which is explained below) will be called.
|
||||
|
||||
:param kwargs: any extra parameters will be passed in a dictionary to any adaptor that receives a parameter called 'adaptor_args'.
|
||||
Check the :ref:`topics-adaptors` topic for more information.
|
||||
|
||||
.. method:: RobustScrapedItem.set_adaptors(self, adaptors_dict)
|
||||
|
||||
Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline.
|
||||
|
||||
.. method:: RobustScrapedItem.set_attrib_adaptors(self, attrib, pipe)
|
||||
|
||||
Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``)
|
||||
|
||||
.. method:: RobustScrapedItem.add_adaptor(self, attrib, adaptor, position=None)
|
||||
|
||||
Adds an adaptor to an already existing (or not) pipeline.
|
||||
|
||||
:param attr: the name of the attribute you're adding adaptors to.
|
||||
|
||||
:param adaptor: a callable to be added to the pipeline.
|
||||
|
||||
:param position: an integer representing the place where to add the adaptor.
|
||||
If it's `None`, the adaptor will be appended at the end of the pipeline.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a pretty basic item with a few attributes::
|
||||
|
||||
from scrapy.contrib.item import RobustScrapedItem
|
||||
|
||||
class MyItem(RobustScrapedItem):
|
||||
ATTRIBUTES = {
|
||||
'name': basestring,
|
||||
'size': basestring,
|
||||
'colours': [basestring],
|
||||
}
|
||||
|
||||
.. note::
|
||||
|
||||
More RobustScrapedItem examples are about to come. In the meantime, check the :ref:`topics-adaptors` topic to see a few of them.
|
||||
|
||||
Now, normally when we're scraping an HTML file, or almost any kind of file, information doesn't come to us exactly as we need it. We usually
|
||||
have to make some adaptations here and there; and that's when the adaptors enter the game.
|
||||
|
|
|
|||
Loading…
Reference in New Issue