10 KiB
Item Exporters
System Message: ERROR/3 (<stdin>, line 7)
Unknown directive type "module".
.. module:: scrapy.contrib.exporter :synopsis: Item Exporters
Once you have scraped your Items, one of the most common tasks to perform on those items is to export them, to use the data in some other application. That is, after all, the whole purpose of the scraping process.
To help in this purpose Scrapy provides a collectioon of Item Exporters for different output formats, such as XML, CSV or JSON.
Using Item Exporters
In order to use a Item Exporter, you must instantiate it with its required args. Different exporters require different args, so check each exporter documentation to be sure, in :ref:`topics-exporters-reference`. After you have instantiated you exporter, you call the method :meth:`~BaseItemExporter.start_exporting` in order to initialize the exporting proces, then you call :meth:`~BaseItemExporter.export_item` method for each item you want to export, and finally call :meth:`~BaseItemExporter.finish_exporting` to finalize the exporting process.
System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "meth".Here you can see a typical Item Exporter usage in an :ref:`Item Pipeline <topics-item-pipeline>`:
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "ref".from scrapy.xlib.pydispatch import dispatcher
from scrapy.contrib.exporter import XmlItemExporter
class XmlExportPipeline(object):
def __init__(self):
dispatcher.connect(self.domain_opened, signals.domain_opened)
dispatcher.connect(self.domain_closed, signals.domain_closed)
def domain_opened(self, domain):
self.file = open('%s_products.xml' % domain)
self.exporter = XmlItemExporter(self.file)
self.exporter.start_exporting()
def domain_closed(self, domain):
self.exporter.finish_exporting()
self.file.close()
def process_item(self, domain, item):
self.exporter.export_item(item)
return item
Serialization of item fields
By default the field values are passed unmodified to the underlying serialization library, and the decision of how to serialize them is delegated to each particular serialization library.
However, you can customize how each field value is serialized, prior to passing it to the serialization library, if the exporter supports it.
There are ways to customize how a field will be serialized, which are described next.
1. Declaring a serializer in the field
You can declare a serializer in the :ref:`field metadata <topics-items-fields>`. The serializer must be a callable which receives a value and returns its serialized form.
System Message: ERROR/3 (<stdin>, line 75); backlink
Unknown interpreted text role "ref".Example:
from scrapy.item import Item, Field
def serialize_price(value):
return '$ %s' % str(value)
class Product(Item):
name = Field()
price = Field(serializer=serialize_price)
2. Overriding the serialize_field() method
You can also override the :meth:`~BaseItemExporter.serialize` method to customize how your field value will be exported.
System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "meth".Make sure you call the base class :meth:`~BaseItemExporter.serialize` method after your custom code.
System Message: ERROR/3 (<stdin>, line 97); backlink
Unknown interpreted text role "meth".Example:
from scrapy.contrib.exporter import XmlItemExporter
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if filed == 'price':
return '$ %s' % str(value)
return super(Product, self).serialize_field(field, name, value)
Built-in Item Exporters reference
For the examples shown in the following exporters we always assume we export these two items:
Item(name='Color TV', price='1200') Item(name='DVD player', price='200')
BaseItemExporter
This is the (abstract) base class for all Item Exporters. It provides support for common features used by all (concrete) Item Exporters, such as defining what fields to export, whether to export empty fields, or which encoding to use.
These features can be configured through the constructor arguments which populate their respective attributes: :attr:`fields_to_export`, :attr:`export_empty_fields`, :attr:`encoding`.
System Message: ERROR/3 (<stdin>, line 132); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 132); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 132); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 136)
Unknown directive type "method".
.. method:: export_item(item) Exports the item to the specific exporter format. This method must be implemented in subclasses.
System Message: ERROR/3 (<stdin>, line 141)
Unknown directive type "method".
.. method:: serialize_field(field, name, value) Return the serialized value for the given field. You can override this method (in your custom Item Exporters) if you want to control how a particular field or value will be serialized/exported. By default, this method looks for a serializer :ref:`declared in the item field <topics-exporters-serializers>` and returns the result of applying that serializer to the value. If no serializer is found, it returns the value unchanged except for ``unicode`` values which are encoded to ``str`` using the encoding declared in the :attr:`encoding` attribute. :param field: the field being serialized :type field: :class:`~scrapy.item.Field` object :param name: the name of the field being serialized :type name: str :param value: the value being serialized
System Message: ERROR/3 (<stdin>, line 161)
Unknown directive type "method".
.. method:: start_exporting() Makes the exporter initialize the export process, in here exporters may output information required by the exporter's format.
System Message: ERROR/3 (<stdin>, line 166)
Unknown directive type "method".
.. method:: finish_exporting() You must call it when there are no more items to export, so the exporter can close the serialization output, for those formats that require it (like XML).
System Message: ERROR/3 (<stdin>, line 172)
Unknown directive type "attribute".
.. attribute:: fields_to_export A list with the name of the fields that will be exported, or None if you want to export all fields. Defaults to None. Some exporters (like :class:`CsvItemExporter`) respect the order of the fields defined in this attribute.
System Message: ERROR/3 (<stdin>, line 180)
Unknown directive type "attribute".
.. attribute:: export_empty_fields Whether to include empty/unpopulated item fields in the exported data. Defaults to ``False``.
System Message: ERROR/3 (<stdin>, line 185)
Unknown directive type "attribute".
.. attribute:: encoding The encoding that will be used to encode unicode values. This only affects unicode values (which are always serialized to str using this encoding). Other value types are passed unchanged to the specific serialization library.
System Message: ERROR/3 (<stdin>, line 192)
Unknown directive type "highlight".
.. highlight:: none
XmlItemExporter
Exports Items in XML format to the specified file object.
| param root_element: | |
|---|---|
| The name of root element in the exported XML. | |
| type root_element: | |
| str | |
| param item_element: | |
| The name of each item element in the exported XML. | |
| type item_element: | |
| str | |
The additional keyword arguments of this constructor are passed to the :class:`BaseItemExporter` constructor.
System Message: ERROR/3 (<stdin>, line 207); backlink
Unknown interpreted text role "class".A typical output of this exporter would be:
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<name>Color TV</name>
<price>1200</price>
</item>
<item>
<name>DVD player</name>
<price>200</price>
</item>
</items>
CsvItemExporter
Exports Items in CSV format to the given file-like object. If the :attr:`fields_to_export` attribute is set, it will be used to define the CSV columns and their order. The :attr:`export_empty_fields` attribute has no effect on this exporter.
System Message: ERROR/3 (<stdin>, line 230); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 230); backlink
Unknown interpreted text role "attr".| param include_headers_line: | |
|---|---|
If enabled, makes the exporter output a header line with the field names taken from :attr:`BaseItemExporter.fields_to_export` so that attribute must also be set in order to work (otherwise it raises a :exc:`RuntimeError`) System Message: ERROR/3 (<stdin>, line 235); backlink Unknown interpreted text role "attr". System Message: ERROR/3 (<stdin>, line 235); backlink Unknown interpreted text role "exc". |
|
| type include_headers_line: | |
boolean |
|
The additional keyword arguments of this constructor are passed to the :class:`BaseItemExporter` constructor, and then to the csv.writer constructor, so you can use any csv.writer constructor argument to customize this exporter.
System Message: ERROR/3 (<stdin>, line 241); backlink
Unknown interpreted text role "class".A typical output of this exporter would be:
Color TV,1200 DVD player,200
PickleItemExporter
Exports Items in pickle format to the given file-like object.
| param protocol: | The pickle protocol to use. |
|---|---|
| type protocol: | int |
For more information, refer to the pickle module documentation.
The additional keyword arguments of this constructor are passed to the :class:`BaseItemExporter` constructor.
System Message: ERROR/3 (<stdin>, line 265); backlink
Unknown interpreted text role "class".This isn't a human readable format, so no output examples are provided.
PprintItemExporter
Exports Items in pretty print format to the specified file object.
The additional keyword arguments of this constructor are passed to the :class:`BaseItemExporter` constructor.
System Message: ERROR/3 (<stdin>, line 279); backlink
Unknown interpreted text role "class".A typical output of this exporter would be:
{'name': 'Color TV', 'price': '1200'}
{'name': 'DVD player', 'price': '200'}
Longer lines (when present) are pretty-formatted.
JsonLinesItemExporter
System Message: ERROR/3 (<stdin>, line 292)
Unknown directive type "module".
.. module:: scrapy.contrib.exporter.jsonlines :synopsis: JsonLines Item Exporter
Exports Items in JSON format to the specified file-like object, writing one JSON-encoded item per line. The additional constructor arguments are passed to the :class:`BaseItemExporter` constructor, and to the JSONEncoder constructor, so you can use any JSONEncoder constructor argument to customize the exporter.
System Message: ERROR/3 (<stdin>, line 297); backlink
Unknown interpreted text role "class".The default output of this exporter would be:
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}