30 KiB

Raw Blame History

Item Loaders

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "module".

.. module:: scrapy.loader
   :synopsis: Item Loader class

Item Loaders provide a convenient mechanism for populating scraped :ref:`Items <topics-items>`. Even though Items can be populated using their own dictionary-like API, Item Loaders provide a much more convenient API for populating them from a scraping process, by automating some common tasks like parsing the raw extracted data before assigning it.

System Message: ERROR/3 (<stdin>, line 10); backlink

Unknown interpreted text role "ref".

In other words, :ref:`Items <topics-items>` provide the container of scraped data, while Item Loaders provide the mechanism for populating that container.

System Message: ERROR/3 (<stdin>, line 16); backlink

Unknown interpreted text role "ref".

Item Loaders are designed to provide a flexible, efficient and easy mechanism for extending and overriding different field parsing rules, either by spider, or by source format (HTML, XML, etc) without becoming a nightmare to maintain.

Using Item Loaders to populate items

To use an Item Loader, you must first instantiate it. You can either instantiate it with a dict-like object (e.g. Item or dict) or without one, in which case an Item is automatically instantiated in the Item Loader constructor using the Item class specified in the :attr:`ItemLoader.default_item_class` attribute.

System Message: ERROR/3 (<stdin>, line 27); backlink

Unknown interpreted text role "attr".

Then, you start collecting values into the Item Loader, typically using :ref:`Selectors <topics-selectors>`. You can add more than one value to the same item field; the Item Loader will know how to "join" those values later using a proper processing function.

System Message: ERROR/3 (<stdin>, line 33); backlink

Unknown interpreted text role "ref".

Here is a typical Item Loader usage in a :ref:`Spider <topics-spiders>`, using the :ref:`Product item <topics-items-declaring>` declared in the :ref:`Items chapter <topics-items>`:

System Message: ERROR/3 (<stdin>, line 38); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 38); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 38); backlink

Unknown interpreted text role "ref".

from scrapy.loader import ItemLoader
from myproject.items import Product

def parse(self, response):
    l = ItemLoader(item=Product(), response=response)
    l.add_xpath('name', '//div[@class="product_name"]')
    l.add_xpath('name', '//div[@class="product_title"]')
    l.add_xpath('price', '//p[@id="price"]')
    l.add_css('stock', 'p#stock]')
    l.add_value('last_updated', 'today') # you can also use literal values
    return l.load_item()

By quickly looking at that code, we can see the name field is being extracted from two different XPath locations in the page:

//div[@class="product_name"]
//div[@class="product_title"]

In other words, data is being collected by extracting it from two XPath locations, using the :meth:`~ItemLoader.add_xpath` method. This is the data that will be assigned to the name field later.

System Message: ERROR/3 (<stdin>, line 60); backlink

Unknown interpreted text role "meth".

Afterwards, similar calls are used for price and stock fields (the latter using a CSS selector with the :meth:`~ItemLoader.add_css` method), and finally the last_update field is populated directly with a literal value (today) using a different method: :meth:`~ItemLoader.add_value`.

System Message: ERROR/3 (<stdin>, line 64); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 64); backlink

Unknown interpreted text role "meth".

Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is called which actually returns the item populated with the data previously extracted and collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls.

System Message: ERROR/3 (<stdin>, line 69); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 69); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 69); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 69); backlink

Unknown interpreted text role "meth".

Input and Output processors

An Item Loader contains one input processor and one output processor for each (item) field. The input processor processes the extracted data as soon as it's received (through the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css` or :meth:`~ItemLoader.add_value` methods) and the result of the input processor is collected and kept inside the ItemLoader. After collecting all data, the :meth:`ItemLoader.load_item` method is called to populate and get the populated :class:`~scrapy.item.Item` object. That's when the output processor is called with the data previously collected (and processed using the input processor). The result of the output processor is the final value that gets assigned to the item.

System Message: ERROR/3 (<stdin>, line 79); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 79); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 79); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 79); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 79); backlink

Unknown interpreted text role "class".

Let's see an example to illustrate how the input and output processors are called for a particular field (the same applies for any other field):

l = ItemLoader(Product(), some_selector)
l.add_xpath('name', xpath1) # (1)
l.add_xpath('name', xpath2) # (2)
l.add_css('name', css) # (3)
l.add_value('name', 'test') # (4)
return l.load_item() # (5)

So what happens is:

Data from xpath1 is extracted, and passed through the input processor of the name field. The result of the input processor is collected and kept in the Item Loader (but not yet assigned to the item).
Data from xpath2 is extracted, and passed through the same input processor used in (1). The result of the input processor is appended to the data collected in (1) (if any).
This case is similar to the previous ones, except that the data is extracted from the css CSS selector, and passed through the same input processor used in (1) and (2). The result of the input processor is appended to the data collected in (1) and (2) (if any).
This case is also similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath expression or a CSS selector. However, the value is still passed through the input processors. In this case, since the value is not iterable it is converted to an iterable of a single element before passing it to the input processor, because input processor always receive iterables.
The data collected in steps (1), (2), (3) and (4) is passed through the output processor of the name field. The result of the output processor is the value assigned to the name field in the item.

It's worth noticing that processors are just callable objects, which are called with the data to be parsed, and return a parsed value. So you can use any function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterator.

Note

Both input and output processors must receive an iterator as their first argument. The output of those functions can be anything. The result of input processors will be appended to an internal list (in the Loader) containing the collected values (for that field). The result of the output processors is the value that will be finally assigned to the item.

If you want to use a plain function as a processor, make sure it receives self as the first argument:

def lowercase_processor(self, values):
    for v in values:
        yield v.lower()

class MyItemLoader(ItemLoader):
    name_in = lowercase_processor

This is because whenever a function is assigned as a class variable, it becomes a method and would be passed the instance as the the first argument when being called. See this answer on stackoverflow for more details.

The other thing you need to keep in mind is that the values returned by input processors are collected internally (in lists) and then passed to output processors to populate the fields.

Last, but not least, Scrapy comes with some :ref:`commonly used processors <topics-loaders-available-processors>` built-in for convenience.

System Message: ERROR/3 (<stdin>, line 157); backlink

Unknown interpreted text role "ref".

Declaring Item Loaders

Item Loaders are declared like Items, by using a class definition syntax. Here is an example:

from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst, MapCompose, Join

class ProductLoader(ItemLoader):

    default_output_processor = TakeFirst()

    name_in = MapCompose(unicode.title)
    name_out = Join()

    price_in = MapCompose(unicode.strip)

    # ...

As you can see, input processors are declared using the _in suffix while output processors are declared using the _out suffix. And you can also declare a default input/output processors using the :attr:`ItemLoader.default_input_processor` and :attr:`ItemLoader.default_output_processor` attributes.

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 182); backlink

Unknown interpreted text role "attr".

Declaring Input and Output Processors

As seen in the previous section, input and output processors can be declared in the Item Loader definition, and it's very common to declare input processors this way. However, there is one more place where you can specify the input and output processors to use: in the :ref:`Item Field <topics-items-fields>` metadata. Here is an example:

System Message: ERROR/3 (<stdin>, line 193); backlink

Unknown interpreted text role "ref".

import scrapy
from scrapy.loader.processors import Join, MapCompose, TakeFirst
from w3lib.html import remove_tags

def filter_price(value):
    if value.isdigit():
        return value

class Product(scrapy.Item):
    name = scrapy.Field(
        input_processor=MapCompose(remove_tags),
        output_processor=Join(),
    )
    price = scrapy.Field(
        input_processor=MapCompose(remove_tags, filter_price),
        output_processor=TakeFirst(),
    )

>>> from scrapy.loader import ItemLoader
>>> il = ItemLoader(item=Product())
>>> il.add_value('name', [u'Welcome to my', u'<strong>website</strong>'])
>>> il.add_value('price', [u'&euro;', u'<span>1000</span>'])
>>> il.load_item()
{'name': u'Welcome to my website', 'price': u'1000'}

The precedence order, for both input and output processors, is as follows:

Item Loader field-specific attributes: field_in and field_out (most precedence)
Field metadata (input_processor and output_processor key)
Item Loader defaults: :meth:`ItemLoader.default_input_processor` and :meth:`ItemLoader.default_output_processor` (least precedence)

System Message: ERROR/3 (<stdin>, line 231); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 231); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 234); backlink

Unknown interpreted text role "ref".

Item Loader Context

The Item Loader Context is a dict of arbitrary key/values which is shared among all input and output processors in the Item Loader. It can be passed when declaring, instantiating or using Item Loader. They are used to modify the behaviour of the input/output processors.

For example, suppose you have a function parse_length which receives a text value and extracts a length from it:

def parse_length(text, loader_context):
    unit = loader_context.get('unit', 'm')
    # ... length parsing code goes here ...
    return parsed_length

By accepting a loader_context argument the function is explicitly telling the Item Loader that it's able to receive an Item Loader context, so the Item Loader passes the currently active context when calling it, and the processor function (parse_length in this case) can thus use them.

There are several ways to modify Item Loader context values:

By modifying the currently active Item Loader context (:attr:`~ItemLoader.context` attribute):

System Message: ERROR/3 (<stdin>, line 261); backlink

Unknown interpreted text role "attr".
```
loader = ItemLoader(product)
loader.context['unit'] = 'cm'
```
On Item Loader instantiation (the keyword arguments of Item Loader constructor are stored in the Item Loader context):
```
loader = ItemLoader(product, unit='cm')
```
On Item Loader declaration, for those input/output processors that support instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of them:

System Message: ERROR/3 (<stdin>, line 272); backlink

Unknown interpreted text role "class".
```
class ProductLoader(ItemLoader):
    length_out = MapCompose(parse_length, unit='cm')
```

ItemLoader objects

Return a new Item Loader for populating the given Item. If no item is given, one is instantiated automatically using the class in :attr:`default_item_class`.

System Message: ERROR/3 (<stdin>, line 285); backlink

Unknown interpreted text role "attr".

When instantiated with a selector or a response parameters the :class:`ItemLoader` class provides convenient mechanisms for extracting data from web pages using :ref:`selectors <topics-selectors>`.

System Message: ERROR/3 (<stdin>, line 289); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 289); backlink

Unknown interpreted text role "ref".

param item:	The item instance to populate using subsequent calls to :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, or :meth:`~ItemLoader.add_value`. System Message: ERROR/3 (`<stdin>`, line 293); backlink Unknown interpreted text role "meth". System Message: ERROR/3 (`<stdin>`, line 293); backlink Unknown interpreted text role "meth". System Message: ERROR/3 (`<stdin>`, line 293); backlink Unknown interpreted text role "meth".
type item:	:class:`~scrapy.item.Item` object System Message: ERROR/3 (`<stdin>`, line 296); backlink Unknown interpreted text role "class".
param selector:	The selector to extract data from, when using the :meth:`add_xpath` (resp. :meth:`add_css`) or :meth:`replace_xpath` (resp. :meth:`replace_css`) method. System Message: ERROR/3 (`<stdin>`, line 298); backlink Unknown interpreted text role "meth". System Message: ERROR/3 (`<stdin>`, line 298); backlink Unknown interpreted text role "meth". System Message: ERROR/3 (`<stdin>`, line 298); backlink Unknown interpreted text role "meth". System Message: ERROR/3 (`<stdin>`, line 298); backlink Unknown interpreted text role "meth".
type selector:	:class:`~scrapy.selector.Selector` object System Message: ERROR/3 (`<stdin>`, line 301); backlink Unknown interpreted text role "class".
param response:	The response used to construct the selector using the :attr:`default_selector_class`, unless the selector argument is given, in which case this argument is ignored. System Message: ERROR/3 (`<stdin>`, line 303); backlink Unknown interpreted text role "attr".
type response:	:class:`~scrapy.http.Response` object System Message: ERROR/3 (`<stdin>`, line 306); backlink Unknown interpreted text role "class".

The item, selector, response and the remaining keyword arguments are assigned to the Loader context (accessible through the :attr:`context` attribute).

System Message: ERROR/3 (<stdin>, line 308); backlink

Unknown interpreted text role "attr".

:class:`ItemLoader` instances have the following methods:

System Message: ERROR/3 (<stdin>, line 311); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 313)