mirror of https://github.com/scrapy/scrapy.git
3.3 KiB
3.3 KiB
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
</head>
</html>
| SEP | 5 |
| Title | ItemBuilder API |
| Author | Ismael Carnales, Pablo Hoffman |
| Created | 2009-07-24 |
| Status | Obsoleted by :doc:`sep-008` System Message: ERROR/3 (<stdin>, line 7); backlink Unknown interpreted text role "doc". |
SEP-005: Detailed ItemBuilder API use
Item class for examples:
#!python
class NewsItem(Item):
url = fields.TextField()
headline = fields.TextField()
content = fields.TextField()
published = fields.DateField()
gSetting expanders
#!python
class NewsItemBuilder(ItemBuilder):
item_class = NewsItem
headline = reducers.Reducer(extract, remove_tags(), unquote(), strip)
This approach will override the Reducer class for BuilderFields depending on their Item Field class:
- MultivaluedField = PassValue
- TextField = JoinStrings
- other = TakeFirst
gSetting reducers
#!python
class NewsItemBuilder(ItemBuilder):
item_class = NewsItem
headline = reducers.TakeFirst(extract, remove_tags(), unquote(), strip)
published = reducers.Reducer(extract, remove_tags(), unquote(), strip)
As with the previous example this would select join_strings as the reducer for content
gSetting expanders/reducers new way
#!python
class NewsItemBuilder(ItemBuilder):
item_class = NewsItem
headline = BuilderField(extract, remove_tags(), unquote(), strip)
content = BuilderField(extract, remove_tags(), unquote(), strip)
class Reducer:
headline = TakeFirst
gExtending ItemBuilder
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(),
strip, to_date('%d.%m.%Y'))
gExtending ItemBuilder using statich methods
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.published, to_date('%d.%m.%Y'))
gUsing default_builder
#!python
class DefaultedNewsItemBuilder(ItemBuilder):
item_class = NewsItem
default_builder = reducers.Reducer(extract, remove_tags(), unquote(), strip)
This will use default_builder as the builder for every field in the item class. As a reducer is not set reducers will be set based on Item Field classes.
gReset default_builder for a field
#!python
class DefaultedNewsItemBuilder(ItemBuilder):
item_class = NewsItem
default_builder = reducers.Reducer(extract, remove_tags(), unquote(), strip)
url = BuilderField()
gExtending default ItemBuilder
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(extract, remove_tags(), unquote(), strip, to_date('%d.%m.%Y'))
gExtending default ItemBuilder using static methods
#!python
class SiteNewsItemBuilder(NewsItemBuilder):
published = reducers.Reducer(NewsItemBuilder.default_builder, to_date('%d.%m.%Y'))