RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/feed-exports.rst

24 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Feed exports

One of the most frequently required features when implementing scrapers is being able to store the scraped data properly and, quite often, that means generating an "export file" with the scraped data (commonly called "export feed") to be consumed by other systems.

Scrapy provides this functionality out of the box with the Feed Exports, which allows you to generate feeds with the scraped items, using multiple serialization formats and storage backends.

This page provides detailed documentation for all feed export features. If you are looking for a step-by-step guide, check out Zyte’s export guides.

Serialization formats

For serializing the scraped data, the feed exports use the :ref:`Item exporters <topics-exporters>`. These formats are supported out of the box:

System Message: ERROR/3 (<stdin>, line 26); backlink

Unknown interpreted text role "ref".

But you can also extend the supported format through the :setting:`FEED_EXPORTERS` setting.

System Message: ERROR/3 (<stdin>, line 34); backlink

Unknown interpreted text role "setting".

JSON

JSON lines

CSV

  • Value for the format key in the :setting:`FEEDS` setting: csv

    System Message: ERROR/3 (<stdin>, line 62); backlink

    Unknown interpreted text role "setting".

  • Exporter used: :class:`~scrapy.exporters.CsvItemExporter`

    System Message: ERROR/3 (<stdin>, line 64); backlink

    Unknown interpreted text role "class".

  • To specify columns to export, their order and their column names, use :setting:`FEED_EXPORT_FIELDS`. Other feed exporters can also use this option, but it is important for CSV because unlike many other export formats CSV uses a fixed header.

    System Message: ERROR/3 (<stdin>, line 66); backlink

    Unknown interpreted text role "setting".

XML

Pickle

Marshal

Storages

When using the feed exports you define where to store the feed using one or multiple URIs (through the :setting:`FEEDS` setting). The feed exports supports multiple storage backend types which are defined by the URI scheme.

System Message: ERROR/3 (<stdin>, line 101); backlink

Unknown interpreted text role "setting".

The storages backends supported out of the box are:

Some storage backends may be unavailable if the required external libraries are not available. For example, the S3 backend is only available if the boto3 library is installed.

Storage URI parameters

The storage URI can also contain parameters that get replaced when the feed is being created. These parameters are:

  • %(time)s - gets replaced by a timestamp when the feed is being created
  • %(name)s - gets replaced by the spider name

Any other named parameter gets replaced by the spider attribute of the same name. For example, %(site_id)s would get replaced by the spider.site_id attribute the moment the feed is being created.

Here are some examples to illustrate:

  • Store in FTP using one directory per spider:
    • ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json
  • Store in S3 using one directory per spider:
    • s3://mybucket/scraping/feeds/%(name)s/%(time)s.json

Note

:ref:`Spider arguments <spiderargs>` become spider attributes, hence they can also be used as storage URI parameters.

System Message: ERROR/3 (<stdin>, line 143); backlink

Unknown interpreted text role "ref".

Note

Only %(...)s parameters are replaced. Any other percent character is kept as-is, so percent-encoded URIs (e.g. %20 for a space or percent-encoded FTP credentials) and :class:`pathlib.Path` keys containing %(...)s parameters both work as expected.

System Message: ERROR/3 (<stdin>, line 146); backlink

Unknown interpreted text role "class".

Storage backends

Local filesystem

The feeds are stored in the local filesystem.

  • URI scheme: file
  • Example URI: file:///tmp/export.csv
  • Required external libraries: none

Note that for the local filesystem storage (only) you can omit the scheme if you specify an absolute path like /tmp/export.csv (Unix systems only). Alternatively you can also use a :class:`pathlib.Path` object.

System Message: ERROR/3 (<stdin>, line 168); backlink

Unknown interpreted text role "class".

FTP

The feeds are stored in a FTP server.

  • URI scheme: ftp
  • Example URI: ftp://user:pass@ftp.example.com/path/to/export.csv
  • Required external libraries: none

FTP supports two different connection modes: active or passive. Scrapy uses the passive connection mode by default. To use the active connection mode instead, set the :setting:`FEED_STORAGE_FTP_ACTIVE` setting to True.

System Message: ERROR/3 (<stdin>, line 183); backlink

Unknown interpreted text role "setting".

The default value for the overwrite key in the :setting:`FEEDS` for this storage backend is: True.

System Message: ERROR/3 (<stdin>, line 188); backlink

Unknown interpreted text role "setting".

Caution!

The value True in overwrite will cause you to lose the previous version of your data.

This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.

System Message: ERROR/3 (<stdin>, line 194); backlink

Unknown interpreted text role "ref".

S3

The feeds are stored on Amazon S3.

  • URI scheme: s3
  • Example URIs:
    • s3://mybucket/path/to/export.csv
    • s3://aws_key:aws_secret@mybucket/path/to/export.csv
  • Required external libraries: boto3 >= 1.20.0

The AWS credentials can be passed as user/password in the URI, or they can be passed through the following settings:

You can also define a custom ACL, custom endpoint, and region name for exported feeds using these settings:

The default value for the overwrite key in the :setting:`FEEDS` for this storage backend is: True.

System Message: ERROR/3 (<stdin>, line 230); backlink

Unknown interpreted text role "setting".

Caution!

The value True in overwrite will cause you to lose the previous version of your data.

This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.

System Message: ERROR/3 (<stdin>, line 236); backlink

Unknown interpreted text role "ref".

Google Cloud Storage (GCS)

The feeds are stored on Google Cloud Storage.

  • URI scheme: gs
  • Example URIs:
    • gs://mybucket/path/to/export.csv
  • Required external libraries: google-cloud-storage.

For more information about authentication, please refer to Google Cloud documentation.

You can set a Project ID and Access Control List (ACL) through the following settings:

The default value for the overwrite key in the :setting:`FEEDS` for this storage backend is: True.

System Message: ERROR/3 (<stdin>, line 261); backlink

Unknown interpreted text role "setting".

Caution!

The value True in overwrite will cause you to lose the previous version of your data.

This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.

System Message: ERROR/3 (<stdin>, line 267); backlink

Unknown interpreted text role "ref".

Standard output

The feeds are written to the standard output of the Scrapy process.

  • URI scheme: stdout
  • Example URI: stdout:
  • Required external libraries: none

Delayed file delivery

As indicated above, some of the described storage backends use delayed file delivery.

These storage backends do not upload items to the feed URI as those items are scraped. Instead, Scrapy writes items into a temporary local file, and only once all the file contents have been written (i.e. at the end of the crawl) is that file uploaded to the feed URI.

If you want item delivery to start earlier when using one of these storage backends, use :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items in multiple files, with the specified maximum item count per file. That way, as soon as a file reaches the maximum item count, that file is delivered to the feed URI, allowing item delivery to start way before the end of the crawl.

System Message: ERROR/3 (<stdin>, line 297); backlink

Unknown interpreted text role "setting".

Item filtering

You can filter items that you want to allow for a particular feed by using the item_classes option in :ref:`feeds options <feed-options>`. Only items of the specified types will be added to the feed.

System Message: ERROR/3 (<stdin>, line 309); backlink

Unknown interpreted text role "ref".

The item_classes option is implemented by the :class:`~scrapy.extensions.feedexport.ItemFilter` class, which is the default value of the item_filter :ref:`feed option <feed-options>`.

System Message: ERROR/3 (<stdin>, line 313); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 313); backlink

Unknown interpreted text role "ref".

You can create your own custom filtering class by implementing :class:`~scrapy.extensions.feedexport.ItemFilter`'s method accepts and taking feed_options as an argument.

System Message: ERROR/3 (<stdin>, line 316); backlink

Unknown interpreted text role "class".

For instance:

System Message: WARNING/2 (<stdin>, line 321)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class MyCustomFilter:
        def __init__(self, feed_options):
            self.feed_options = feed_options

        def accepts(self, item):
            if "field1" in item and item["field1"] == "expected_data":
                return True
            return False

You can assign your custom filtering class to the item_filter :ref:`option of a feed <feed-options>`. See :setting:`FEEDS` for examples.

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 333); backlink

Unknown interpreted text role "setting".

ItemFilter

System Message: ERROR/3 (<stdin>, line 339)

Unknown directive type "autoclass".

.. autoclass:: scrapy.extensions.feedexport.ItemFilter
   :members:

Post-Processing

Scrapy provides an option to activate plugins to post-process feeds before they are exported to feed storages. In addition to using :ref:`builtin plugins <builtin-plugins>`, you can create your own :ref:`plugins <custom-plugins>`.

System Message: ERROR/3 (<stdin>, line 348); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 348); backlink

Unknown interpreted text role "ref".

These plugins can be activated through the postprocessing option of a feed. The option must be passed a list of post-processing plugins in the order you want the feed to be processed. These plugins can be declared either as an import string or with the imported class of the plugin. Parameters to plugins can be passed through the feed options. See :ref:`feed options <feed-options>` for examples.

System Message: ERROR/3 (<stdin>, line 352); backlink

Unknown interpreted text role "ref".

Built-in Plugins

System Message: ERROR/3 (<stdin>, line 363)

Unknown directive type "autoclass".

.. autoclass:: scrapy.extensions.postprocessing.GzipPlugin

System Message: ERROR/3 (<stdin>, line 365)

Unknown directive type "autoclass".

.. autoclass:: scrapy.extensions.postprocessing.LZMAPlugin

System Message: ERROR/3 (<stdin>, line 367)

Unknown directive type "autoclass".

.. autoclass:: scrapy.extensions.postprocessing.Bz2Plugin

Custom Plugins

Each plugin is a class that must implement the following methods:

System Message: ERROR/3 (<stdin>, line 376)

Unknown directive type "method".

.. method:: __init__(self, file, feed_options)

    Initialize the plugin.

    :param file: file-like object having at least the `write`, `tell` and `close` methods implemented

    :param feed_options: feed-specific :ref:`options <feed-options>`
    :type feed_options: :class:`dict`

System Message: ERROR/3 (<stdin>, line 385)

Unknown directive type "method".

.. method:: write(self, data)

   Process and write `data` (:class:`bytes` or :class:`memoryview`) into the plugin's target file.
   It must return number of bytes written.

System Message: ERROR/3 (<stdin>, line 390)

Unknown directive type "method".

.. method:: close(self)

    Clean up the plugin.

    For example, you might want to close a file wrapper that you might have
    used to compress data written into the file received in the ``__init__``
    method.

    .. warning:: Do not close the file from the ``__init__`` method.

To pass a parameter to your plugin, use :ref:`feed options <feed-options>`. You can then access those parameters from the __init__ method of your plugin.

System Message: ERROR/3 (<stdin>, line 400); backlink

Unknown interpreted text role "ref".

Settings

These are the settings used for configuring the feed exports:

System Message: ERROR/3 (<stdin>, line 420)

Unknown directive type "currentmodule".

.. currentmodule:: scrapy.extensions.feedexport

System Message: ERROR/3 (<stdin>, line 422)

Unknown directive type "setting".

.. setting:: FEEDS

FEEDS

Default: {}

A dictionary in which every key is a feed URI (or a :class:`pathlib.Path` object) and each value is a nested dictionary containing configuration parameters for the specific feed.

System Message: ERROR/3 (<stdin>, line 429); backlink

Unknown interpreted text role "class".

This setting is required for enabling the feed export feature.

See :ref:`topics-feed-storage-backends` for supported URI schemes.

System Message: ERROR/3 (<stdin>, line 435); backlink

Unknown interpreted text role "ref".

For instance:

{
    'items.json': {
        'format': 'json',
        'encoding': 'utf8',
        'store_empty': False,
        'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'],
        'fields': None,
        'indent': 4,
        'item_export_kwargs': {
           'export_empty_fields': True,
        },
    },
    '/home/user/documents/items.xml': {
        'format': 'xml',
        'fields': ['name', 'price'],
        'item_filter': MyCustomFilter1,
        'encoding': 'latin1',
        'indent': 8,
    },
    pathlib.Path('items.csv.gz'): {
        'format': 'csv',
        'fields': ['price', 'name'],
        'item_filter': 'myproject.filters.MyCustomFilter2',
        'postprocessing': [MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'],
        'gzip_compresslevel': 5,
    },
}

The following is a list of the accepted keys and the setting that is used as a fallback value if that key is not provided for a specific feed definition:

System Message: ERROR/3 (<stdin>, line 522)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_ENCODING

FEED_EXPORT_ENCODING

Default: "utf-8" (:ref:`fallback <default-settings>`: None)

System Message: ERROR/3 (<stdin>, line 527); backlink

Unknown interpreted text role "ref".

The encoding to be used for the feed.

If set to None, it uses UTF-8 for everything except JSON output, which uses safe numeric encoding (\uXXXX sequences) for historic reasons.

Use "utf-8" if you want UTF-8 for JSON too.

System Message: ERROR/3 (<stdin>, line 536)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_FIELDS

FEED_EXPORT_FIELDS

Default: None

Use the FEED_EXPORT_FIELDS setting to define the fields to export, their order and their output names. See :attr:`BaseItemExporter.fields_to_export <scrapy.exporters.BaseItemExporter.fields_to_export>` for more information.

System Message: ERROR/3 (<stdin>, line 543); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 547)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_INDENT

FEED_EXPORT_INDENT

Default: 0

Amount of spaces used to indent the output on each level. If FEED_EXPORT_INDENT is a non-negative integer, then array elements and object members will be pretty-printed with that indent level. An indent level of 0 (the default), or negative, will put each item on a new line. None selects the most compact representation.

Currently implemented only by :class:`~scrapy.exporters.JsonItemExporter` and :class:`~scrapy.exporters.XmlItemExporter`, i.e. when you are exporting to .json or .xml.

System Message: ERROR/3 (<stdin>, line 559); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 559); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 563)

Unknown directive type "setting".

.. setting:: FEED_STORE_EMPTY

FEED_STORE_EMPTY

Default: True

Whether to export empty feeds (i.e. feeds with no items). If False, and there are no items to export, no new files are created and existing files are not modified, even if the :ref:`overwrite feed option <feed-options>` is enabled.

System Message: ERROR/3 (<stdin>, line 570); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 575)

Unknown directive type "setting".

.. setting:: FEED_STORAGES

FEED_STORAGES

Default: {}

A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes.

System Message: ERROR/3 (<stdin>, line 585)

Unknown directive type "setting".

.. setting:: FEED_STORAGE_FTP_ACTIVE

FEED_STORAGE_FTP_ACTIVE

Default: False

Whether to use the active connection mode when exporting feeds to an FTP server (True) or use the passive connection mode instead (False, default).

For information about FTP connection modes, see What is the difference between active and passive FTP?.

System Message: ERROR/3 (<stdin>, line 598)

Unknown directive type "setting".

.. setting:: FEED_STORAGE_S3_ACL

FEED_STORAGE_S3_ACL

Default: '' (empty string)

A string containing a custom ACL for feeds exported to Amazon S3 by your project.

For a complete list of available values, access the Canned ACL section on Amazon S3 docs.

System Message: ERROR/3 (<stdin>, line 609)

Unknown directive type "setting".

.. setting:: FEED_STORAGES_BASE

FEED_STORAGES_BASE

Default:

System Message: WARNING/2 (<stdin>, line 616)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    {
        "": "scrapy.extensions.feedexport.FileFeedStorage",
        "file": "scrapy.extensions.feedexport.FileFeedStorage",
        "stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
        "s3": "scrapy.extensions.feedexport.S3FeedStorage",
        "gs": "scrapy.extensions.feedexport.GCSFeedStorage",
        "ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
    }

A dict containing the built-in feed storage backends supported by Scrapy. You can disable any of these backends by assigning None to their URI scheme in :setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend (without replacement), place this in your settings.py:

System Message: ERROR/3 (<stdin>, line 627); backlink

Unknown interpreted text role "setting".

System Message: WARNING/2 (<stdin>, line 632)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    FEED_STORAGES = {
        "ftp": None,
    }

System Message: ERROR/3 (<stdin>, line 638)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS

FEED_EXPORTERS

Default: {}

A dict containing additional exporters supported by your project. The keys are serialization formats and the values are paths to :ref:`Item exporter <topics-exporters>` classes.

System Message: ERROR/3 (<stdin>, line 645); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 649)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS_BASE

FEED_EXPORTERS_BASE

Default:

System Message: WARNING/2 (<stdin>, line 655)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    {
        "json": "scrapy.exporters.JsonItemExporter",
        "jsonlines": "scrapy.exporters.JsonLinesItemExporter",
        "jsonl": "scrapy.exporters.JsonLinesItemExporter",
        "jl": "scrapy.exporters.JsonLinesItemExporter",
        "csv": "scrapy.exporters.CsvItemExporter",
        "xml": "scrapy.exporters.XmlItemExporter",
        "marshal": "scrapy.exporters.MarshalItemExporter",
        "pickle": "scrapy.exporters.PickleItemExporter",
    }

A dict containing the built-in feed exporters supported by Scrapy. You can disable any of these exporters by assigning None to their serialization format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter (without replacement), place this in your settings.py:

System Message: ERROR/3 (<stdin>, line 668); backlink

Unknown interpreted text role "setting".

System Message: WARNING/2 (<stdin>, line 673)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    FEED_EXPORTERS = {
        "csv": None,
    }

System Message: ERROR/3 (<stdin>, line 680)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_BATCH_ITEM_COUNT

FEED_EXPORT_BATCH_ITEM_COUNT

Default: 0

If assigned an integer number higher than 0, Scrapy generates multiple output files storing up to the specified number of items in each output file.

When generating multiple output files, you must use at least one of the following placeholders in the feed URI to indicate how the different output file names are generated:

  • %(batch_time)s - gets replaced by a timestamp when the feed is being created (e.g. 2020-03-28T14-45-08.237134)

  • %(batch_id)d - gets replaced by the 1-based sequence number of the batch.

    Use :ref:`printf-style string formatting <python:old-string-formatting>` to alter the number format. For example, to make the batch ID a 5-digit number by introducing leading zeroes as needed, use %(batch_id)05d (e.g. 3 becomes 00003, 123 becomes 00123).

    System Message: ERROR/3 (<stdin>, line 699); backlink

    Unknown interpreted text role "ref".

For instance, if your settings include:

System Message: WARNING/2 (<stdin>, line 706)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    FEED_EXPORT_BATCH_ITEM_COUNT = 100

And your :command:`crawl` command line is:

System Message: ERROR/3 (<stdin>, line 710); backlink

Unknown interpreted text role "command".
scrapy crawl spidername -o "dirname/%(batch_id)d-filename%(batch_time)s.json"

The command line above can generate a directory tree like:

->projectname
-->dirname
--->1-filename2020-03-28T14-45-08.237134.json
--->2-filename2020-03-28T14-45-09.148903.json
--->3-filename2020-03-28T14-45-10.046092.json

Where the first and second files contain exactly 100 items. The last one contains 100 items or fewer.

System Message: ERROR/3 (<stdin>, line 726)

Unknown directive type "setting".

.. setting:: FEED_URI_PARAMS

FEED_URI_PARAMS

Default: None

A string with the import path of a function to set the parameters to apply with :ref:`printf-style string formatting <python:old-string-formatting>` to the feed URI.

System Message: ERROR/3 (<stdin>, line 733); backlink

Unknown interpreted text role "ref".

The function signature should be as follows:

System Message: ERROR/3 (<stdin>, line 739)

Unknown directive type "function".

.. function:: uri_params(params, spider)

   Return a :class:`dict` of key-value pairs to apply to the feed URI using
   :ref:`printf-style string formatting <python:old-string-formatting>`.

   :param params: default key-value pairs

        Specifically:

        -   ``batch_id``: ID of the file batch. See
            :setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.

            If :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` is ``0``, ``batch_id``
            is always ``1``.

        -   ``batch_time``: UTC date and time, in ISO format with ``:``
            replaced with ``-``.

            See :setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.

        -   ``time``: ``batch_time``, with microseconds set to ``0``.
   :type params: dict

   :param spider: source spider of the feed items
   :type spider: scrapy.Spider

   .. caution:: The function should return a new dictionary, modifying
                the received ``params`` in-place is deprecated.

For example, to include the :attr:`name <scrapy.Spider.name>` of the source spider in the feed URI:

System Message: ERROR/3 (<stdin>, line 768); backlink

Unknown interpreted text role "attr".

  1. Define the following function somewhere in your project:

    System Message: WARNING/2 (<stdin>, line 773)

    Cannot analyze code. Pygments package not found.

    .. code-block:: python

    # myproject/utils.py
    def uri_params(params, spider):
        return {**params, "spider_name": spider.name}

  2. Point :setting:`FEED_URI_PARAMS` to that function in your settings:

    System Message: ERROR/3 (<stdin>, line 779); backlink

    Unknown interpreted text role "setting".

    System Message: WARNING/2 (<stdin>, line 781)

    Cannot analyze code. Pygments package not found.

    .. code-block:: python

    # myproject/settings.py
    FEED_URI_PARAMS = "myproject.utils.uri_params"

  3. Use %(spider_name)s in your feed URI:

    scrapy crawl <spider_name> -o "%(spider_name)s.jsonl"
</html>