diff --git a/docs/experimental/exporters.rst b/docs/experimental/exporters.rst new file mode 100644 index 000000000..2f664434b --- /dev/null +++ b/docs/experimental/exporters.rst @@ -0,0 +1,178 @@ +.. _topics-exporters: + +============== +Item Exporters +============== + +.. module:: scrapy.contrib.exporter + :synopsis: Item Exporters + +Once you have scraped your Items, you probably will want to use the data in some +external application. For this purpose Scrapy provides simple Item Exporters +that allow you to export your scraped Items in different formats. + + +Using Item Exporters +==================== + +In order to use a Item Exporter, you must instantiate it with its required args +(different exporters require different args in order to work, look at the +reference of the specific Item Exporter in :ref:`topics-exporters-reference` ) +then call the :meth:`~BaseItemExporter.export` method with the item to export as +its argument. + +Here you can see a typical Item Exporter usage in a :ref:`Item Pipeline +`:: + + from scrapy.xlib.pydispatch import dispatcher + from scrapy.contrib.exporter.jsonexporter import JSONItemExporter + + + class ProductJSONPipeline(object): + def __init__(self): + dispatcher.connect(self.domain_open, signals.domain_open) + dispatcher.connect(self.domain_closed, signals.domain_closed) + + def domain_open(self, domain): + self.file = open('%s_products.json' % domain) + self.exporter = JSONItemExporter(self.file) + + def domain_closed(self, domain): + self.file.close() + + def process_item(self, domain, item): + self.exporter.export(item) + return item + + +.. _topics-exporters-field-serialization: + +Field serialization +=================== + +By default each field is serialized to its string representation using the +:meth:`~BaseItemExporter._default_serializer` method. + +You can customize how a field will be serialized in two ways: + +1. Providing a ``serialize_(field-name)`` method in your custom Item Exporter. +2. Implementing a ``serializer`` method in a custom Field of your Item. + +.. note:: This is the order of precedence, so if you provide both, the custom + serialize_(field-name) method in the Item Exporter will be used. + +In any case, your method must accept the same parameters as the +:meth:`~BaseItemExporter._default_serializer` method and return the serialized +version of the field in a string format. + +Let's see some examples on using the methods described above: + +1. Providing a ``serialize_price`` method:: + + from scrapy.contrib.exporter.jsonexporter import JSONItemExporter + + class ProductJSONExporter(JSONItemExporter): + def serialize_price(self, field, name, value): + return '$ %s' % str(value) + +2. Using a custom ``PriceField``:: + + from scrapy.newitem import Item, Field + + class PriceField(Field): + def serializer(self, field, name, value): + return '$ %s' % str(value) + + class Product(Item): + name = Field() + price = PriceField() + stock = Field(default=0) + last_updated = Field() + + +.. _topics-exporters-reference: + +Available Item Exporters +======================== + +BaseItemExporter +---------------- + +.. class:: BaseItemExporter + + This is the base class for all Item Exporters. + + .. method:: export(item) + + Exports the item to the specific exporter format. Descendant classes must + override this method. + + .. method:: _default_serializer(field, name, value) + + Serializes the field, the base implementation returns it string + representation. You can override this in custom Item Exporters. + + +PprintItemExporter +------------------ + +.. class:: PprintItemExporter(file) + + Exports Items in preety print format to the specified file object. + + +.. class:: PickleItemExporter(\*args, \**kwargs) + + Exports Items in pickle format. The arguments in the constructor will be used + to construct a ``cPickle.Pickler`` object. + + +CsvItemExporter +--------------- + +.. class:: CsvItemExporter(\*args, \**kwargs) + + Exports Items in CSV format. The arguments in the constructor will be used to + construct a ``csv.writer`` object. You must also set its + :attr:`~CsvItemExporter.fields_to_export` attribute to use it. + + .. attribute:: fields_to_export + + Iterable containing the Item Field names to be exported. + + +XmlItemExporter +--------------- + +.. class:: XmlItemExporter(\*args, \**kwargs) + + Exports Items in XML format to the specified file object. You must also set its + :attr:`~XmlItemExporter.fields_to_export` attribute to use it. + + .. attribute:: root_element + + The name of the root element in the exported file. It defaults to + ``items``. + + .. attribute:: item_element + + The name of each item element in the exported file. It defaults to + ``item``. + + .. attribute:: include_empty_elements + + Whether to include or not empty elements in the exported file. It defaults to + ``False``. + + .. attribute:: fields_to_export + + Iterable containing the Item Field names to be exported. + + +JSONItemExporter +---------------- + +.. class:: scrapy.contrib.exporter.jsonexporter.JsonItemExporter(file) + + Exports Items in JSON format to the specified file object. + diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst index 55468737d..8f4e33489 100644 --- a/docs/experimental/index.rst +++ b/docs/experimental/index.rst @@ -21,3 +21,4 @@ it's properly merged) . Use at your own risk. newitems loaders + exporters