mirror of https://github.com/scrapy/scrapy.git
179 lines
5.0 KiB
ReStructuredText
179 lines
5.0 KiB
ReStructuredText
.. _topics-exporters:
|
|
|
|
==============
|
|
Item Exporters
|
|
==============
|
|
|
|
.. module:: scrapy.contrib.exporter
|
|
:synopsis: Item Exporters
|
|
|
|
Once you have scraped your Items, you probably will want to use the data in some
|
|
external application. For this purpose Scrapy provides simple Item Exporters
|
|
that allow you to export your scraped Items in different formats.
|
|
|
|
|
|
Using Item Exporters
|
|
====================
|
|
|
|
In order to use a Item Exporter, you must instantiate it with its required args
|
|
(different exporters require different args in order to work, look at the
|
|
reference of the specific Item Exporter in :ref:`topics-exporters-reference` )
|
|
then call the :meth:`~BaseItemExporter.export` method with the item to export as
|
|
its argument.
|
|
|
|
Here you can see a typical Item Exporter usage in a :ref:`Item Pipeline
|
|
<topics-item-pipeline>`::
|
|
|
|
from scrapy.xlib.pydispatch import dispatcher
|
|
from scrapy.contrib.exporter.jsonexporter import JSONItemExporter
|
|
|
|
|
|
class ProductJSONPipeline(object):
|
|
def __init__(self):
|
|
dispatcher.connect(self.domain_open, signals.domain_open)
|
|
dispatcher.connect(self.domain_closed, signals.domain_closed)
|
|
|
|
def domain_open(self, domain):
|
|
self.file = open('%s_products.json' % domain)
|
|
self.exporter = JSONItemExporter(self.file)
|
|
|
|
def domain_closed(self, domain):
|
|
self.file.close()
|
|
|
|
def process_item(self, domain, item):
|
|
self.exporter.export(item)
|
|
return item
|
|
|
|
|
|
.. _topics-exporters-field-serialization:
|
|
|
|
Field serialization
|
|
===================
|
|
|
|
By default each field is serialized to its string representation using the
|
|
:meth:`~BaseItemExporter._default_serializer` method.
|
|
|
|
You can customize how a field will be serialized in two ways:
|
|
|
|
1. Providing a ``serialize_(field-name)`` method in your custom Item Exporter.
|
|
2. Implementing a ``serializer`` method in a custom Field of your Item.
|
|
|
|
.. note:: This is the order of precedence, so if you provide both, the custom
|
|
serialize_(field-name) method in the Item Exporter will be used.
|
|
|
|
In any case, your method must accept the same parameters as the
|
|
:meth:`~BaseItemExporter._default_serializer` method and return the serialized
|
|
version of the field in a string format.
|
|
|
|
Let's see some examples on using the methods described above:
|
|
|
|
1. Providing a ``serialize_price`` method::
|
|
|
|
from scrapy.contrib.exporter.jsonexporter import JSONItemExporter
|
|
|
|
class ProductJSONExporter(JSONItemExporter):
|
|
def serialize_price(self, field, name, value):
|
|
return '$ %s' % str(value)
|
|
|
|
2. Using a custom ``PriceField``::
|
|
|
|
from scrapy.newitem import Item, Field
|
|
|
|
class PriceField(Field):
|
|
def serializer(self, field, name, value):
|
|
return '$ %s' % str(value)
|
|
|
|
class Product(Item):
|
|
name = Field()
|
|
price = PriceField()
|
|
stock = Field(default=0)
|
|
last_updated = Field()
|
|
|
|
|
|
.. _topics-exporters-reference:
|
|
|
|
Available Item Exporters
|
|
========================
|
|
|
|
BaseItemExporter
|
|
----------------
|
|
|
|
.. class:: BaseItemExporter
|
|
|
|
This is the base class for all Item Exporters.
|
|
|
|
.. method:: export(item)
|
|
|
|
Exports the item to the specific exporter format. Descendant classes must
|
|
override this method.
|
|
|
|
.. method:: _default_serializer(field, name, value)
|
|
|
|
Serializes the field, the base implementation returns it string
|
|
representation. You can override this in custom Item Exporters.
|
|
|
|
|
|
PprintItemExporter
|
|
------------------
|
|
|
|
.. class:: PprintItemExporter(file)
|
|
|
|
Exports Items in preety print format to the specified file object.
|
|
|
|
|
|
.. class:: PickleItemExporter(\*args, \**kwargs)
|
|
|
|
Exports Items in pickle format. The arguments in the constructor will be used
|
|
to construct a ``cPickle.Pickler`` object.
|
|
|
|
|
|
CsvItemExporter
|
|
---------------
|
|
|
|
.. class:: CsvItemExporter(\*args, \**kwargs)
|
|
|
|
Exports Items in CSV format. The arguments in the constructor will be used to
|
|
construct a ``csv.writer`` object. You must also set its
|
|
:attr:`~CsvItemExporter.fields_to_export` attribute to use it.
|
|
|
|
.. attribute:: fields_to_export
|
|
|
|
Iterable containing the Item Field names to be exported.
|
|
|
|
|
|
XmlItemExporter
|
|
---------------
|
|
|
|
.. class:: XmlItemExporter(\*args, \**kwargs)
|
|
|
|
Exports Items in XML format to the specified file object. You must also set its
|
|
:attr:`~XmlItemExporter.fields_to_export` attribute to use it.
|
|
|
|
.. attribute:: root_element
|
|
|
|
The name of the root element in the exported file. It defaults to
|
|
``items``.
|
|
|
|
.. attribute:: item_element
|
|
|
|
The name of each item element in the exported file. It defaults to
|
|
``item``.
|
|
|
|
.. attribute:: include_empty_elements
|
|
|
|
Whether to include or not empty elements in the exported file. It defaults to
|
|
``False``.
|
|
|
|
.. attribute:: fields_to_export
|
|
|
|
Iterable containing the Item Field names to be exported.
|
|
|
|
|
|
JSONItemExporter
|
|
----------------
|
|
|
|
.. class:: scrapy.contrib.exporter.jsonexporter.JsonItemExporter(file)
|
|
|
|
Exports Items in JSON format to the specified file object.
|
|
|