mirror of https://github.com/scrapy/scrapy.git
removed obsolete documentation about Robust Scraped Item and Adaptors
This commit is contained in:
Pablo Hoffman
parent
bc4ec80db7
commit
d4aa72d2bb
|
|
@ -1,205 +0,0 @@
|
|||
.. _topics-adaptors:
|
||||
|
||||
==============================
|
||||
Old Item Adaptors (deprecated)
|
||||
==============================
|
||||
|
||||
.. warning::
|
||||
|
||||
This documentation is deprecated, and will be superseeded by the new item
|
||||
adaptors (not yet documented).
|
||||
|
||||
Quick overview
|
||||
==============
|
||||
|
||||
Scrapy's adaptors are a nice feature attached to :class:`RobustScrapedItem`
|
||||
that allow you to easily modify (adapt to your needs) any kind of information
|
||||
you want to put in your items at assignation time.
|
||||
|
||||
The following diagram shows the data flow from the moment you call the
|
||||
``attribute`` method until the attribute is actually set.
|
||||
|
||||
.. image:: _images/adaptors_diagram.png
|
||||
|
||||
As you can see, adaptor pipelines are executed in tree form; which means that,
|
||||
for each of the values you pass to the ``attribute`` method, the first adaptor
|
||||
will be applied. Then, for each of the resulting values of the first adaptor,
|
||||
the second adaptor will be called, and so on. This process will end up with a
|
||||
list of adapted values, which may contain zero, one, or many values.
|
||||
|
||||
In case the attribute is a single-valued (this is defined in the item's
|
||||
``ATTRIBUTES`` dictionary), the first element of this list will be set, unless
|
||||
you call the ``attribute`` method with the add parameter as True, in which case
|
||||
the item's method ``_add_single_attributes`` will be called with the
|
||||
attribute's name, type, and the list of attributes to join as parameters. By
|
||||
default, this method raises NotImplementedError, so you should override it in
|
||||
your items in order to join any kind of objects.
|
||||
|
||||
If the attribute is a multivalued, the resulting list will be set to the item
|
||||
as is, unless you use -again- add=True, in which case the list of
|
||||
already-existing values (if any) will be extended with the new one.pgq
|
||||
|
||||
Adaptor Pipelines
|
||||
=================
|
||||
|
||||
.. class:: AdaptorPipe(adaptors=None)
|
||||
|
||||
An instance of this class represents an adaptor pipeline to be set for
|
||||
adapting a certain item's attribute. It provides some useful methods for
|
||||
adding/removing adaptors, and takes care of executing them properly.
|
||||
Usually this class is not used directly, since the items already provide
|
||||
ways to manage adaptors without having to handle AdaptorPipes.
|
||||
|
||||
:param adaptors: A list of callables to be added as adaptors at
|
||||
instancing time.
|
||||
|
||||
Methods:
|
||||
|
||||
.. method:: add_adaptor(adaptor, position=None)
|
||||
|
||||
This method is used for adding adaptors to the pipeline given
|
||||
a certain position.
|
||||
|
||||
:param adaptor: Any callable that works as an adaptor
|
||||
:param position: An integer meaning the position in which the adaptor
|
||||
will be inserted. If it's None the adaptor will be appended at
|
||||
the end of the pipeline.
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
As it was previously said, in order to use adaptor pipelines you must inherit
|
||||
your items from the :class:`RobustScrapedItem` class. If you don't know
|
||||
anything about these items, read the :ref:`topics-items` reference first.
|
||||
|
||||
Once you've created your own item class (inherited from
|
||||
:class:`RobustScrapedItem`) with the attributes you're going to use, you have
|
||||
to add adaptor pipelines to each attribute you'd like to adapt data for. For
|
||||
doing so, RobustScrapedItems provide some useful methods like ``set_adaptors``,
|
||||
``set_attrib_adaptors``, and more (which are also described in its reference)
|
||||
so that you don't need to work with :class:`AdaptorPipe` objects directly.
|
||||
|
||||
Adaptors
|
||||
--------
|
||||
|
||||
Let's now talk a bit about adaptors (singularly), what are them, and how
|
||||
should they be implemented?
|
||||
|
||||
Adaptors are basically, any callable that receives
|
||||
a value, modifies it, and returns a new value (or more) so that the next
|
||||
adaptor goes on with another adapting task (or not). This is done this way to
|
||||
make the process of modifying information very customizable, and also to make
|
||||
adaptors reusable, since they are intended to be small functions designed for
|
||||
simple purposes that can be applied in many different cases. For example, you
|
||||
could make an adaptor for removing any <b> tags in a text, like this::
|
||||
|
||||
>>> B_TAG_RE = re.compile(r'</?b\s*>')
|
||||
>>> def remove_b_tags(text):
|
||||
>>> return B_TAG_RE.sub('', text)
|
||||
|
||||
Then you could easily add this adaptor to a certain attribute's pipeline like
|
||||
this::
|
||||
|
||||
>>> item = MyItem()
|
||||
>>> item.add_adaptor('text', remove_b_tags)
|
||||
>>> item.attribute('text', u'<b>some random text in bold</b> and some random text in normal font')
|
||||
>>> item.text
|
||||
u'some random text in bold and some random text in normal font'
|
||||
|
||||
As you can see, this would make any value that you set to the item through the
|
||||
``attribute`` method first pass through the ``remove_b_tags`` adaptor, which
|
||||
would also replace any matching tag with an empty string.
|
||||
|
||||
----
|
||||
|
||||
But anyway, let's now think of a bit more complicated (and useless) example:
|
||||
let's say you want to scrape a text, split it into single letters, strip the
|
||||
vowels, turn the rest to capital letters, and join them again. In this case,
|
||||
we could use three simple adaptors to process our data, plus a customized
|
||||
:class:`RobustScrapedItem` for joining single text attributes; let's see an
|
||||
example::
|
||||
|
||||
>>> # First of all, we define the item class we're going to use
|
||||
>>> from string import ascii_letters
|
||||
>>> from scrapy.contrib.item import RobustScrapedItem
|
||||
>>> class MyItem(RobustScrapedItem):
|
||||
>>> ATTRIBUTES = {
|
||||
>>> 'text': basestring,
|
||||
>>> }
|
||||
|
||||
>>> def _add_single_attributes(self, attrname, attrtype, attributes):
|
||||
>>> return ''.join(attributes)
|
||||
|
||||
>>> # Now we'll write the needed adaptors
|
||||
>>> def to_letters(text):
|
||||
>>> return tuple(letter for letter in text)
|
||||
|
||||
>>> def is_vowel(letter):
|
||||
>>> if letter in ascii_letters and letter.lower() not in ('a', 'e', 'i', 'o', 'u'):
|
||||
>>> return letter
|
||||
|
||||
>>> def to_upper(letter):
|
||||
>>> return letter.upper()
|
||||
|
||||
>>> # Finally, we'll join all the pieces and see how it works
|
||||
>>> item = MyItem()
|
||||
>>> item.set_attrib_adaptors('text', [
|
||||
>>> to_letters,
|
||||
>>> is_vowel,
|
||||
>>> to_upper,
|
||||
>>> ])
|
||||
|
||||
Let's now try with an example text to see what happens::
|
||||
|
||||
>>> item.attribute('text', 'pi', 'wind', add=True)
|
||||
>>> item.text
|
||||
'PWND'
|
||||
|
||||
More complex adaptors
|
||||
---------------------
|
||||
|
||||
Now, after using adaptors a bit, you may find yourself in situations where you need
|
||||
to use adaptors that receive other parameters from the ``attribute`` method
|
||||
apart from the value to adapt.
|
||||
|
||||
For example, imagine you have an adaptor that removes certain characters from strings
|
||||
you provide. Would you make an adaptor for each combination of characters you'd like
|
||||
to strip? Of course not!
|
||||
|
||||
The way to handle this cases, is to make an adaptor that apart from receiving a value,
|
||||
as any other adaptor, receives a parameter called ``adaptor_args``.
|
||||
It's important that the parameter is called this way, since Scrapy finds out whether
|
||||
an adaptor is able to receive extra parameters or not by making instrospection
|
||||
and looking for a parameter called this way in the adaptor's parameters list.
|
||||
|
||||
The information this parameter will receive won't be anything else but the same dictionary
|
||||
of keyword arguments that you pass to the ``attribute`` method when calling it.
|
||||
|
||||
But let's get back to the characters example, how would we implement this?
|
||||
Quite simmilar to any other adaptor, let's see::
|
||||
|
||||
def strip_chars(value, adaptor_args):
|
||||
chars = adaptor_args.get('strip_chars', [])
|
||||
for char in chars:
|
||||
value = value.replace(char, '')
|
||||
return value
|
||||
|
||||
Then, after creating an item and adding the adaptor to one of its pipelines, we could do::
|
||||
|
||||
>>> item.attribute('text', 'Hi, my name is John', strip_chars=['a', 'i', 'm'])
|
||||
>>> item.text
|
||||
'H, y ne s John'
|
||||
|
||||
Debugging
|
||||
=========
|
||||
|
||||
While you're coding spiders and adaptors, you usually need to know exactly what
|
||||
does Scrapy do under the hood with the values you provide. There's a setting
|
||||
called :setting:``ADAPTORS_DEBUG`` for this purpose that makes Scrapy print
|
||||
debugging messages each time an adaptors pipeline is run, specifying which
|
||||
attribute is being adapted data for, the input/output values of each adaptor in
|
||||
the pipeline, and the input/output of ``_add_single_attributes`` (in some
|
||||
cases).
|
||||
|
||||
You can enable this setting as any other, either by adding it to your settings
|
||||
file, or by enabling the environment variable ``SCRAPY_ADAPTORS_DEBUG``.
|
||||
|
|
@ -51,144 +51,3 @@ Creating an item and setting its attributes inline::
|
|||
>>> person
|
||||
ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'})
|
||||
|
||||
RobustScrapedItems
|
||||
==================
|
||||
|
||||
.. warning::
|
||||
|
||||
RobustScapedItems are deprecated and will be replaced by the :ref:`New item
|
||||
API <topics-newitems>` (still in development).
|
||||
|
||||
.. module:: scrapy.contrib.item
|
||||
:synopsis: Objects for storing scraped data
|
||||
|
||||
.. class:: RobustScrapedItem
|
||||
|
||||
RobustScrapedItems are more complex items (compared to
|
||||
:class:`ScrapedItem`) and have a few more features available, which
|
||||
include:
|
||||
|
||||
* Attributes dictionary: items that inherit from RobustScrapedItem are
|
||||
defined with a dictionary of attributes in the class. This allows the
|
||||
item to have more logic at the moment of handling and setting attributes
|
||||
than the :class:`ScrapedItem`.
|
||||
|
||||
* Adaptors: perhaps the most important of the features these items provide.
|
||||
The adaptors are a system designed for filtering/modifying data before
|
||||
setting it to the item, that makes cleansing tasks a lot easier.
|
||||
|
||||
* Type checking: RobustScrapedItems come with a built-in type checking
|
||||
which assures you that no data of the wrong type will get into the items
|
||||
without raising a warning.
|
||||
|
||||
* Versioning: These items also provide versioning by making a unique hash
|
||||
for each item based on its attributes values.
|
||||
|
||||
* ItemDeltas: You can subtract two RobustScrapedItems, which allows you to
|
||||
know the difference between a pair of items. This difference is
|
||||
represented by a RobustItemDelta object.
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: RobustScrapedItem.ATTRIBUTES
|
||||
|
||||
This attribute **must** be specified when writing your items, and it's a
|
||||
dictionary in which the keys are the names of the attributes your item will
|
||||
have, and their values are the type of those attributes. For multivalued
|
||||
attributes, you should write the type of the values inside a list, e.g:
|
||||
``'numbers': [int]``
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: RobustScrapedItem.__init__(data=None, adaptor_args=None)
|
||||
|
||||
:param data: Idem as for ScrapedItems
|
||||
:param adaptor_args: A dictionary of the kind
|
||||
``'attribute': [list_of_adaptors]``" for defining adaptors automatically
|
||||
after instancing the item.
|
||||
|
||||
Constructor of RobustScrapedItem objects.
|
||||
|
||||
.. method:: RobustScrapedItem.attribute(attrname, value, override=False, add=False, ***kwargs)
|
||||
|
||||
Sets the item's ``attrname`` attribute with the given ``value`` filtering
|
||||
it through the given attribute's adaptor pipeline (if any).
|
||||
|
||||
:param attrname: a string containing the name of the attribute you want
|
||||
to set.
|
||||
|
||||
:param value: the value you want to assign, which will be adapted by
|
||||
the corresponding adaptors for the given attribute (if any).
|
||||
|
||||
:param override: if True, makes this method avoid checking if there
|
||||
was a previous value and sets ``value`` no matter what.
|
||||
|
||||
:param add: if True, tries to concatenate the given ``value`` with the one
|
||||
already set in the item. For multivalued attributes, this will extend
|
||||
the list of already-set values, with the new ones.
|
||||
For single valued attributes, the method _add_single_attributes (which
|
||||
is explained below) will be called.
|
||||
|
||||
:param kwargs: any extra parameters will be passed in a dictionary to any
|
||||
adaptor that receives a parameter called ``adaptor_args``.
|
||||
Check the :ref:`topics-adaptors` topic for more information.
|
||||
|
||||
.. method:: RobustScrapedItem.set_adaptors(adaptors_dict)
|
||||
|
||||
Receives a dict containing a list of adaptors for each desired attribute
|
||||
(key) and sets each of them as their adaptor pipeline.
|
||||
|
||||
.. method:: RobustScrapedItem.set_attrib_adaptors(attrib, pipe)
|
||||
|
||||
Sets the provided iterable (``pipe``) as the adaptor pipeline for the
|
||||
given attribute (``attrib``)
|
||||
|
||||
.. method:: RobustScrapedItem.add_adaptor(attrib, adaptor, position=None)
|
||||
|
||||
Adds an adaptor to an already existing (or not) pipeline.
|
||||
|
||||
:param attr: the name of the attribute you're adding adaptors to.
|
||||
|
||||
:param adaptor: a callable to be added to the pipeline.
|
||||
|
||||
:param position: an integer representing the place where to add the adaptor.
|
||||
If it's ``None``, the adaptor will be appended at the end of the pipeline.
|
||||
|
||||
.. method:: RobustScrapedItem._add_single_attributes(attrname, attrtype, attributes)
|
||||
|
||||
This method is the one to be called whenever a single attribute has to be
|
||||
joined before storing into an item. That is,
|
||||
every time you have multiple results at the end of your adaptors pipeline,
|
||||
and you called the ``attribute`` method with the parameter `add=True`.
|
||||
|
||||
This method is intended to be overriden by you, since by default it
|
||||
raises an exception.
|
||||
|
||||
:param attrname: the name of the attribute you're setting
|
||||
:param attrtype: the type of the attribute you're setting
|
||||
:param attributes: the list of resulting values after the adaptors pipeline
|
||||
(the one you have to join somehow)
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Creating a pretty basic item with a few attributes::
|
||||
|
||||
from scrapy.contrib.item import RobustScrapedItem
|
||||
|
||||
class MyItem(RobustScrapedItem):
|
||||
ATTRIBUTES = {
|
||||
'name': basestring,
|
||||
'size': basestring,
|
||||
'colours': [basestring],
|
||||
}
|
||||
|
||||
Setting some adaptors::
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
More RobustScrapedItem examples are about to come. In the meantime, check the :ref:`topics-adaptors` topic to see a few of them.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue