18 KiB
Item Exporters
System Message: ERROR/3 (<stdin>, line 7)
Unknown directive type "module".
.. module:: scrapy.exporters :synopsis: Item Exporters
Once you have scraped your items, you often want to persist or export those items, to use the data in some other application. That is, after all, the whole purpose of the scraping process.
For this purpose Scrapy provides a collection of Item Exporters for different output formats, such as XML, CSV or JSON.
Using Item Exporters
If you are in a hurry, and just want to use an Item Exporter to output scraped data see the :ref:`topics-feed-exports`. Otherwise, if you want to know how Item Exporters work or need more custom functionality (not covered by the default exports), continue reading below.
System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "ref".In order to use an Item Exporter, you must instantiate it with its required args. Each Item Exporter requires different arguments, so check each exporter documentation to be sure, in :ref:`topics-exporters-reference`. After you have instantiated your exporter, you have to:
System Message: ERROR/3 (<stdin>, line 25); backlink
Unknown interpreted text role "ref".1. call the method :meth:`~BaseItemExporter.start_exporting` in order to signal the beginning of the exporting process
System Message: ERROR/3 (<stdin>, line 30); backlink
Unknown interpreted text role "meth".2. call the :meth:`~BaseItemExporter.export_item` method for each item you want to export
System Message: ERROR/3 (<stdin>, line 33); backlink
Unknown interpreted text role "meth".3. and finally call the :meth:`~BaseItemExporter.finish_exporting` to signal the end of the exporting process
System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "meth".Here you can see an :doc:`Item Pipeline <item-pipeline>` which uses multiple Item Exporters to group scraped items to different files according to the value of one of their fields:
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "doc".System Message: WARNING/2 (<stdin>, line 43)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from itemadapter import ItemAdapter
from scrapy.exporters import XmlItemExporter
class PerYearXmlExportPipeline:
"""Distribute items across multiple XML files according to their 'year' field"""
def open_spider(self, spider):
self.year_to_exporter = {}
def close_spider(self, spider):
for exporter, xml_file in self.year_to_exporter.values():
exporter.finish_exporting()
xml_file.close()
def _exporter_for_item(self, item):
adapter = ItemAdapter(item)
year = adapter["year"]
if year not in self.year_to_exporter:
xml_file = open(f"{year}.xml", "wb")
exporter = XmlItemExporter(xml_file)
exporter.start_exporting()
self.year_to_exporter[year] = (exporter, xml_file)
return self.year_to_exporter[year][0]
def process_item(self, item):
exporter = self._exporter_for_item(item)
exporter.export_item(item)
return item
Serialization of item fields
By default, the field values are passed unmodified to the underlying serialization library, and the decision of how to serialize them is delegated to each particular serialization library.
However, you can customize how each field value is serialized before it is passed to the serialization library.
There are two ways to customize how a field will be serialized, which are described next.
1. Declaring a serializer in the field
Every :ref:`item type <item-types>` except :class:`dict` lets you declare a serializer in the :ref:`field metadata <topics-items-fields>`. The serializer must be a callable which receives a value and returns its serialized form.
System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "ref".Example:
System Message: WARNING/2 (<stdin>, line 102)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from dataclasses import dataclass, field
def serialize_price(value):
return f"$ {str(value)}"
@dataclass
class Product:
name: str
price: float = field(metadata={"serializer": serialize_price})
2. Overriding the serialize_field() method
You can also override the :meth:`~BaseItemExporter.serialize_field` method to customize how your field value will be exported.
System Message: ERROR/3 (<stdin>, line 120); backlink
Unknown interpreted text role "meth".Make sure you call the base class :meth:`~BaseItemExporter.serialize_field` method after your custom code.
System Message: ERROR/3 (<stdin>, line 123); backlink
Unknown interpreted text role "meth".Example:
System Message: WARNING/2 (<stdin>, line 128)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy.exporters import XmlItemExporter
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if name == "price":
return f"$ {str(value)}"
return super().serialize_field(field, name, value)
Writing your own item exporter
To write an item exporter, subclass :class:`BaseItemExporter` and implement :meth:`~BaseItemExporter.export_item`, where :meth:`~BaseItemExporter.get_serialized_fields` gives you the (name, value) pairs to export.
System Message: ERROR/3 (<stdin>, line 144); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 144); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 144); backlink
Unknown interpreted text role "meth".To make your exporter available to the :ref:`feed exports <topics-feed-exports>`, list it in the :setting:`FEED_EXPORTERS` setting. Feed exports :ref:`build <from-crawler>` it with the output file as the first positional argument, and with the fields, encoding and indent :ref:`feed options <feed-options>` and every key of item_export_kwargs as keyword arguments, so your __init__ method must forward unknown keyword arguments to :class:`BaseItemExporter`.
System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "class".The file object belongs to whoever opened it, i.e. to the feed storage in the case of feed exports, which also closes it. If you need a text file, for example to use :func:`csv.writer` or another Python API that does not accept a binary file, wrap it with :class:`io.TextIOWrapper` and call :meth:`~io.TextIOBase.detach` on the wrapper in :meth:`~BaseItemExporter.finish_exporting`; otherwise the wrapper closes the underlying file when it is garbage-collected.
System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "meth".For example, the following item exporter writes items as blocks of name: value lines:
System Message: WARNING/2 (<stdin>, line 168)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from io import TextIOWrapper
from scrapy.exporters import BaseItemExporter
class TextItemExporter(BaseItemExporter):
def __init__(self, file, item_separator="\n", **kwargs):
super().__init__(**kwargs)
self.item_separator = item_separator
self.stream = TextIOWrapper(
file, encoding=self.encoding or "utf-8", write_through=True
)
def export_item(self, item):
for name, value in self.get_serialized_fields(item):
print(f"{name}: {value}", file=self.stream)
self.stream.write(self.item_separator)
def finish_exporting(self):
self.stream.detach()
To use it as the txt feed format:
System Message: WARNING/2 (<stdin>, line 193)
Cannot analyze code. Pygments package not found.
.. code-block:: python
FEED_EXPORTERS = {"txt": "myproject.exporters.TextItemExporter"}
FEEDS = {
"items.txt": {
"format": "txt",
"item_export_kwargs": {"item_separator": "---\n"},
},
}
Built-in Item Exporters reference
Here is a list of the Item Exporters bundled with Scrapy. Some of them contain output examples, which assume you're exporting these two items:
System Message: WARNING/2 (<stdin>, line 212)
Cannot analyze code. Pygments package not found.
.. code-block:: python
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
BaseItemExporter
This is the (abstract) base class for all Item Exporters. It provides support for common features used by all (concrete) Item Exporters, such as defining what fields to export, whether to export empty fields, or which encoding to use.
These features can be configured through the __init__ method arguments which populate their respective instance attributes: :attr:`fields_to_export`, :attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`.
System Message: ERROR/3 (<stdin>, line 227); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 227); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 227); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 227); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 231)
Unknown directive type "method".
.. method:: export_item(item) Exports the given item. This method must be implemented in subclasses.
System Message: ERROR/3 (<stdin>, line 235)
Unknown directive type "automethod".
.. automethod:: BaseItemExporter.get_serialized_fields
System Message: ERROR/3 (<stdin>, line 237)
Unknown directive type "method".
.. method:: serialize_field(field, name, value)
Return the serialized value for the given field. You can override this
method (in your custom Item Exporters) if you want to control how a
particular field or value will be serialized/exported.
By default, this method looks for a serializer :ref:`declared in the item
field <topics-exporters-serializers>` and returns the result of applying
that serializer to the value. If no serializer is found, it returns the
value unchanged.
:param field: the field being serialized. If the source :ref:`item object
<item-types>` does not define field metadata, *field* is an empty
:class:`dict`.
:type field: :class:`~scrapy.Field` object or a :class:`dict` instance
:param name: the name of the field being serialized
:type name: str
:param value: the value being serialized
System Message: ERROR/3 (<stdin>, line 258)
Unknown directive type "method".
.. method:: start_exporting() Signal the beginning of the exporting process. Some exporters may use this to generate some required header (for example, the :class:`XmlItemExporter`). You must call this method before exporting any items.
System Message: ERROR/3 (<stdin>, line 265)
Unknown directive type "method".
.. method:: finish_exporting() Signal the end of the exporting process. Some exporters may use this to generate some required footer (for example, the :class:`XmlItemExporter`). You must always call this method after you have no more items to export.
System Message: ERROR/3 (<stdin>, line 272)
Unknown directive type "attribute".
.. attribute:: fields_to_export
Fields to export, their order [1]_ and their output names.
Possible values are:
- ``None`` (all fields [2]_, default)
Fields are exported in declaration order, i.e. the order in which
they are defined in the :ref:`item class <item-types>`. For
:class:`dict` items, which have no declared fields, the key order of
each item is used instead.
.. versionchanged:: VERSION
Fields of non-\ :class:`dict` items used to be exported in the
order in which they had been populated, except in
:class:`CsvItemExporter`, which has always used declaration order.
- A list of fields:
.. code-block:: python
["field1", "field2"]
- A dict where keys are fields and values are output names:
.. code-block:: python
{"field1": "Field 1", "field2": "Field 2"}
.. [1] Not all exporters respect the specified field order.
.. [2] When using :ref:`item objects <item-types>` that do not expose
all their possible fields, exporters that do not support exporting
a different subset of fields per item will only export the fields
found in the first item exported.
System Message: ERROR/3 (<stdin>, line 308)
Unknown directive type "attribute".
.. attribute:: export_empty_fields Whether to include empty/unpopulated item fields in the exported data. Defaults to ``False``. Some exporters (like :class:`CsvItemExporter`) ignore this attribute and always export all empty fields. This option is ignored for dict items.
System Message: ERROR/3 (<stdin>, line 316)
Unknown directive type "attribute".
.. attribute:: encoding The output character encoding.
System Message: ERROR/3 (<stdin>, line 320)
Unknown directive type "attribute".
.. attribute:: indent
Amount of spaces used to indent the output on each level. Defaults to ``None``.
* ``indent=None`` selects the most compact representation,
all items in the same line with no indentation
* ``indent<=0`` each item on its own line, no indentation
* ``indent>0`` each item on its own line, indented with the provided numeric value
PythonItemExporter
System Message: ERROR/3 (<stdin>, line 332)
Unknown directive type "autoclass".
.. autoclass:: PythonItemExporter
System Message: ERROR/3 (<stdin>, line 335)
Unknown directive type "highlight".
.. highlight:: none
XmlItemExporter
Exports items in XML format to the specified file object.
| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|---|
| param root_element: | |
| The name of root element in the exported XML. | |
| type root_element: | |
| str | |
| param item_element: | |
| The name of each item element in the exported XML. | |
| type item_element: | |
| str | |
The additional keyword arguments of this __init__ method are passed to the :class:`BaseItemExporter` __init__ method.
System Message: ERROR/3 (<stdin>, line 353); backlink
Unknown interpreted text role "class".A typical output of this exporter would be:
System Message: WARNING/2 (<stdin>, line 358)
Cannot analyze code. Pygments package not found.
.. code-block:: xml
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<name>Color TV</name>
<price>1200</price>
</item>
<item>
<name>DVD player</name>
<price>200</price>
</item>
</items>
Unless overridden in the :meth:`serialize_field` method, multi-valued fields are exported by serializing each value inside a <value> element. This is for convenience, as multi-valued fields are very common.
System Message: ERROR/3 (<stdin>, line 372); backlink
Unknown interpreted text role "meth".For example, the item:
System Message: WARNING/2 (<stdin>, line 380)
Cannot analyze code. Pygments package not found.
.. code-block:: python
Item(name=["John", "Doe"], age="23")
Would be serialized as:
System Message: WARNING/2 (<stdin>, line 386)
Cannot analyze code. Pygments package not found.
.. code-block:: xml
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<name>
<value>John</value>
<value>Doe</value>
</name>
<age>23</age>
</item>
</items>
CsvItemExporter
Exports items in CSV format to the given file-like object. If the :attr:`fields_to_export` attribute is set, it will be used to define the CSV columns, their order and their column names. The :attr:`export_empty_fields` attribute has no effect on this exporter.
System Message: ERROR/3 (<stdin>, line 404); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 404); backlink
Unknown interpreted text role "attr".| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|---|
| param include_headers_line: | |
If enabled, makes the exporter output a header line with the field names taken from :attr:`BaseItemExporter.fields_to_export` or the first exported item fields. System Message: ERROR/3 (<stdin>, line 412); backlink Unknown interpreted text role "attr". |
|
| type include_headers_line: | |
bool |
|
| param join_multivalued: | |
The char (or chars) that will be used for joining multi-valued fields, if found. |
|
| type join_multivalued: | |
str |
|
| param errors: | The optional string that specifies how encoding and decoding errors are to be handled. For more information see :class:`io.TextIOWrapper`. System Message: ERROR/3 (<stdin>, line 421); backlink Unknown interpreted text role "class". |
| type errors: | str |
The additional keyword arguments of this __init__ method are passed to the :class:`BaseItemExporter` __init__ method, and the leftover arguments to the :func:`csv.writer` function, so you can use any :func:`csv.writer` function argument to customize this exporter.
System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 426); backlink
Unknown interpreted text role "func".A typical output of this exporter would be:
name,price Color TV,1200 DVD player,200
PickleItemExporter
Exports items in pickle format to the given file-like object.
| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|---|
| param protocol: | The pickle protocol to use. |
| type protocol: | int |
For more information, see :mod:`pickle`.
System Message: ERROR/3 (<stdin>, line 450); backlink
Unknown interpreted text role "mod".The additional keyword arguments of this __init__ method are passed to the :class:`BaseItemExporter` __init__ method.
System Message: ERROR/3 (<stdin>, line 452); backlink
Unknown interpreted text role "class".Pickle isn't a human readable format, so no output examples are provided.
PprintItemExporter
Exports items in pretty print format to the specified file object.
| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|
The additional keyword arguments of this __init__ method are passed to the :class:`BaseItemExporter` __init__ method.
System Message: ERROR/3 (<stdin>, line 467); backlink
Unknown interpreted text role "class".A typical output of this exporter would be:
System Message: WARNING/2 (<stdin>, line 472)
Cannot analyze code. Pygments package not found.
.. code-block:: python
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
Longer lines (when present) are pretty-formatted.
JsonItemExporter
Exports items in JSON format to the specified file-like object, writing all objects as a list of objects. The additional __init__ method arguments are passed to the :class:`BaseItemExporter` __init__ method, and the leftover arguments to the :class:`~json.JSONEncoder` __init__ method, so you can use any :class:`~json.JSONEncoder` __init__ method argument to customize this exporter.
System Message: ERROR/3 (<stdin>, line 484); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 484); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 484); backlink
Unknown interpreted text role "class".| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|
A typical output of this exporter would be:
System Message: WARNING/2 (<stdin>, line 495)
Cannot analyze code. Pygments package not found.
.. code-block:: json
[{"name": "Color TV", "price": "1200"},
{"name": "DVD player", "price": "200"}]
Warning
JSON is very simple and flexible serialization format, but it doesn't scale well for large amounts of data since incremental (aka. stream-mode) parsing is not well supported (if at all) among JSON parsers (on any language), and most of them just parse the entire object in memory. If you want the power and simplicity of JSON with a more stream-friendly format, consider using :class:`JsonLinesItemExporter` instead, or splitting the output in multiple chunks.
System Message: ERROR/3 (<stdin>, line 502); backlink
Unknown interpreted text role "class".JsonLinesItemExporter
Exports items in JSON format to the specified file-like object, writing one JSON-encoded item per line. The additional __init__ method arguments are passed to the :class:`BaseItemExporter` __init__ method, and the leftover arguments to the :class:`~json.JSONEncoder` __init__ method, so you can use any :class:`~json.JSONEncoder` __init__ method argument to customize this exporter.
System Message: ERROR/3 (<stdin>, line 515); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 515); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 515); backlink
Unknown interpreted text role "class".| param file: | the file-like object to use for exporting the data. Its write method should accept bytes (a disk file opened in binary mode, a io.BytesIO object, etc) |
|---|
A typical output of this exporter would be:
System Message: WARNING/2 (<stdin>, line 526)
Cannot analyze code. Pygments package not found.
.. code-block:: json
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
Unlike the one produced by :class:`JsonItemExporter`, the format produced by this exporter is well suited for serializing large amounts of data.
System Message: ERROR/3 (<stdin>, line 531); backlink
Unknown interpreted text role "class".MarshalItemExporter
System Message: ERROR/3 (<stdin>, line 537)
Unknown directive type "autoclass".
.. autoclass:: MarshalItemExporter