RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/experimental/exporters.rst

5.0 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Item Exporters

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "module".

.. module:: scrapy.contrib.exporter
   :synopsis: Item Exporters

Once you have scraped your Items, you probably will want to use the data in some external application. For this purpose Scrapy provides simple Item Exporters that allow you to export your scraped Items in different formats.

Using Item Exporters

In order to use a Item Exporter, you must instantiate it with its required args (different exporters require different args in order to work, look at the reference of the specific Item Exporter in :ref:`topics-exporters-reference` ) then call the :meth:`~BaseItemExporter.export` method with the item to export as its argument.

System Message: ERROR/3 (<stdin>, line 18); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 18); backlink

Unknown interpreted text role "meth".

Here you can see a typical Item Exporter usage in a :ref:`Item Pipeline <topics-item-pipeline>`:

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "ref".
from scrapy.xlib.pydispatch import dispatcher
from scrapy.contrib.exporter.jsonexporter import JSONItemExporter


class ProductJSONPipeline(object):
    def __init__(self):
        dispatcher.connect(self.domain_open, signals.domain_open)
        dispatcher.connect(self.domain_closed, signals.domain_closed)

    def domain_open(self, domain):
        self.file = open('%s_products.json' % domain)
        self.exporter = JSONItemExporter(self.file)

    def domain_closed(self, domain):
        self.file.close()

    def process_item(self, domain, item):
        self.exporter.export(item)
        return item

Field serialization

By default each field is serialized to its string representation using the :meth:`~BaseItemExporter._default_serializer` method.

System Message: ERROR/3 (<stdin>, line 53); backlink

Unknown interpreted text role "meth".

You can customize how a field will be serialized in two ways:

  1. Providing a serialize_(field-name) method in your custom Item Exporter.
  2. Implementing a serializer method in a custom Field of your Item.

Note

This is the order of precedence, so if you provide both, the custom serialize_(field-name) method in the Item Exporter will be used.

In any case, your method must accept the same parameters as the :meth:`~BaseItemExporter._default_serializer` method and return the serialized version of the field in a string format.

System Message: ERROR/3 (<stdin>, line 64); backlink

Unknown interpreted text role "meth".

Let's see some examples on using the methods described above:

  1. Providing a serialize_price method:

    from scrapy.contrib.exporter.jsonexporter import JSONItemExporter

class ProductJSONExporter(JSONItemExporter):
    def serialize_price(self, field, name, value):
        return '$ %s' % str(value)

  2. Using a custom PriceField:

    from scrapy.newitem import Item, Field

class PriceField(Field):
   def serializer(self, field, name, value):
       return '$ %s' % str(value)

class Product(Item):
    name = Field()
    price = PriceField()
    stock = Field(default=0)
    last_updated = Field()

Available Item Exporters

BaseItemExporter

This is the base class for all Item Exporters.

System Message: ERROR/3 (<stdin>, line 105)

Unknown directive type "method".

.. method:: export(item)

   Exports the item to the specific exporter format. Descendant classes must
   override this method.

System Message: ERROR/3 (<stdin>, line 110)

Unknown directive type "method".

.. method:: _default_serializer(field, name, value)

   Serializes the field, the base implementation returns it string
   representation. You can override this in custom Item Exporters.

PprintItemExporter

Exports Items in preety print format to the specified file object.

Exports Items in pickle format. The arguments in the constructor will be used to construct a cPickle.Pickler object.

CsvItemExporter

Exports Items in CSV format. The arguments in the constructor will be used to construct a csv.writer object. You must also set its :attr:`~CsvItemExporter.fields_to_export` attribute to use it.

System Message: ERROR/3 (<stdin>, line 135); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 139)

Unknown directive type "attribute".

.. attribute:: fields_to_export

   Iterable containing the Item Field names to be exported.

XmlItemExporter

Exports Items in XML format to the specified file object. You must also set its :attr:`~XmlItemExporter.fields_to_export` attribute to use it.

System Message: ERROR/3 (<stdin>, line 149); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 152)

Unknown directive type "attribute".

.. attribute:: root_element

   The name of the root element in the exported file. It defaults to
   ``items``.

System Message: ERROR/3 (<stdin>, line 157)

Unknown directive type "attribute".

.. attribute:: item_element

   The name of each item element in the exported file. It defaults to
   ``item``.

System Message: ERROR/3 (<stdin>, line 162)

Unknown directive type "attribute".

.. attribute:: include_empty_elements

   Whether to include or not empty elements in the exported file. It defaults to
   ``False``.

System Message: ERROR/3 (<stdin>, line 167)

Unknown directive type "attribute".

.. attribute:: fields_to_export

   Iterable containing the Item Field names to be exported.

JSONItemExporter

Exports Items in JSON format to the specified file object.

</html>