scrapy/sep/sep-005.rst

3.5 KiB

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>
SEP 5
Title ItemBuilder API
Author Ismael Carnales, Pablo Hoffman
Created 2009-07-24
Status

Obsoleted by :doc:`sep-008`

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "doc".

SEP-005: Detailed ItemBuilder API use

Item class for examples:

System Message: WARNING/2 (<stdin>, line 15)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class NewsItem(Item):
       url = fields.TextField()
       headline = fields.TextField()
       content = fields.TextField()
       published = fields.DateField()


gSetting expanders

System Message: WARNING/2 (<stdin>, line 28)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class NewsItemBuilder(ItemBuilder):
       item_class = NewsItem

       headline = reducers.Reducer(extract, remove_tags(), unquote(), strip)


This approach will override the Reducer class for BuilderFields depending on their Item Field class:

  • MultivaluedField = PassValue
  • TextField = JoinStrings
  • other = TakeFirst

gSetting reducers

System Message: WARNING/2 (<stdin>, line 47)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class NewsItemBuilder(ItemBuilder):
       item_class = NewsItem

       headline = reducers.TakeFirst(extract, remove_tags(), unquote(), strip)
       published = reducers.Reducer(extract, remove_tags(), unquote(), strip)


As with the previous example this would select join_strings as the reducer for content

gSetting expanders/reducers new way

System Message: WARNING/2 (<stdin>, line 63)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class NewsItemBuilder(ItemBuilder):
       item_class = NewsItem

       headline = BuilderField(extract, remove_tags(), unquote(), strip)
       content = BuilderField(extract, remove_tags(), unquote(), strip)

       class Reducer:
           headline = TakeFirst


gExtending ItemBuilder

System Message: WARNING/2 (<stdin>, line 79)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class SiteNewsItemBuilder(NewsItemBuilder):
       published = reducers.Reducer(
           extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
       )


gExtending ItemBuilder using statich methods

System Message: WARNING/2 (<stdin>, line 91)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class SiteNewsItemBuilder(NewsItemBuilder):
       published = reducers.Reducer(NewsItemBuilder.published, to_date("%d.%m.%Y"))


gUsing default_builder

System Message: WARNING/2 (<stdin>, line 101)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class DefaultedNewsItemBuilder(ItemBuilder):
       item_class = NewsItem

       default_builder = reducers.Reducer(extract, remove_tags(), unquote(), strip)


This will use default_builder as the builder for every field in the item class. As a reducer is not set reducers will be set based on Item Field classes.

gReset default_builder for a field

System Message: WARNING/2 (<stdin>, line 116)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class DefaultedNewsItemBuilder(ItemBuilder):
       item_class = NewsItem

       default_builder = reducers.Reducer(extract, remove_tags(), unquote(), strip)
       url = BuilderField()


gExtending default ItemBuilder

System Message: WARNING/2 (<stdin>, line 129)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class SiteNewsItemBuilder(NewsItemBuilder):
       published = reducers.Reducer(
           extract, remove_tags(), unquote(), strip, to_date("%d.%m.%Y")
       )


gExtending default ItemBuilder using static methods

System Message: WARNING/2 (<stdin>, line 141)

Cannot analyze code. Pygments package not found.

.. code-block:: python

   #!python
   class SiteNewsItemBuilder(NewsItemBuilder):
       published = reducers.Reducer(NewsItemBuilder.default_builder, to_date("%d.%m.%Y"))
</html>