3.4 KiB
| SEP | 8 |
| Title | Item Parsers |
| Author | Pablo Hoffman |
| Created | 2009-08-11 |
| Status | Final (implemented with variations) |
| Obsoletes | :doc:`sep-001`, :doc:`sep-002`, :doc:`sep-003`, :doc:`sep-005` System Message: ERROR/3 (<stdin>, line 8); backlink Unknown interpreted text role "doc".System Message: ERROR/3 (<stdin>, line 8); backlink Unknown interpreted text role "doc".System Message: ERROR/3 (<stdin>, line 8); backlink Unknown interpreted text role "doc".System Message: ERROR/3 (<stdin>, line 8); backlink Unknown interpreted text role "doc". |
SEP-008 - Item Loaders
Item Parser is the final API proposed to implement Item Builders/Loader proposed in :doc:`sep-001`.
System Message: ERROR/3 (<stdin>, line 14); backlink
Unknown interpreted text role "doc".Note
This is the API that was finally implemented with the name "Item Loaders", instead of "Item Parsers" along with some other minor fine tuning to the API methods and semantics.
Dataflow
- ItemParser.add_value() 1. input_parser 2. store
- ItemParser.add_xpath() (only available in XPathItemLoader) 1. selector.extract() 2. input_parser 3. store
- ItemParser.populate_item() (ex. get_item) 1. output_parser 2. assign field
Modules and classes
- scrapy.contrib.itemparser.ItemParser
- scrapy.contrib.itemparser.XPathItemParser
- scrapy.contrib.itemparser.parsers.MapConcat (ex. ``TreeExpander``)
- scrapy.contrib.itemparser.parsers.TakeFirst
- scrapy.contrib.itemparser.parsers.Join
- scrapy.contrib.itemparser.parsers.Identity
Public API
- ItemParser.add_value()
- ItemParser.replace_value()
- ItemParser.populate_item() (returns item populated)
- ItemParser.get_collected_values() (note the 's' in values)
- ItemParser.parse_field()
- ItemParser.get_input_parser()
- ItemParser.get_output_parser()
- ItemParser.context
- ItemParser.default_item_class
- ItemParser.default_input_parser
- ItemParser.default_output_parser
- ItemParser.*field*_in
- ItemParser.*field*_out
Alternative Public API Proposal
- ItemLoader.add_value()
- ItemLoader.replace_value()
- ItemLoader.load_item() (returns loaded item)
- ItemLoader.get_stored_values() or ItemLoader.get_values() (returns the ``ItemLoader`` values)
- ItemLoader.get_output_value()
- ItemLoader.get_input_processor() or ItemLoader.get_in_processor() (short version)
- ItemLoader.get_output_processor() or ItemLoader.get_out_processor() (short version)
- ItemLoader.context
- ItemLoader.default_item_class
- ItemLoader.default_input_processor or ItemLoader.default_in_processor (short version)
- ItemLoader.default_output_processor or ItemLoader.default_out_processor (short version)
- ItemLoader.*field*_in
- ItemLoader.*field*_out
Usage example: declaring Item Parsers
System Message: WARNING/2 (<stdin>, line 90)
Cannot analyze code. Pygments package not found.
.. code-block:: python
#!python
from scrapy.contrib.itemparser import XPathItemParser, parsers
class ProductParser(XPathItemParser):
name_in = parsers.MapConcat(removetags, filterx)
price_in = parsers.MapConcat(...)
price_out = parsers.TakeFirst()
Usage example: declaring parsers in Fields
System Message: WARNING/2 (<stdin>, line 105)
Cannot analyze code. Pygments package not found.
.. code-block:: python
#!python
class Product(Item):
name = Field(output_parser=parsers.Join(), ...)
price = Field(output_parser=parsers.TakeFirst(), ...)
description = Field(input_parser=parsers.MapConcat(removetags))