RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/feed-exports.rst

12 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Feed exports

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "versionadded".

.. versionadded:: 0.10

One of the most frequently required features when implementing scrapers is being able to store the scraped data properly and, quite often, that means generating an "export file" with the scraped data (commonly called "export feed") to be consumed by other systems.

Scrapy provides this functionality out of the box with the Feed Exports, which allows you to generate feeds with the scraped items, using multiple serialization formats and storage backends.

Serialization formats

For serializing the scraped data, the feed exports use the :ref:`Item exporters <topics-exporters>`. These formats are supported out of the box:

System Message: ERROR/3 (<stdin>, line 23); backlink

Unknown interpreted text role "ref".

But you can also extend the supported format through the :setting:`FEED_EXPORTERS` setting.

System Message: ERROR/3 (<stdin>, line 31); backlink

Unknown interpreted text role "setting".

JSON

JSON lines

CSV

  • Value for the format key in the :setting:`FEEDS` setting: csv

    System Message: ERROR/3 (<stdin>, line 57); backlink

    Unknown interpreted text role "setting".

  • Exporter used: :class:`~scrapy.exporters.CsvItemExporter`

    System Message: ERROR/3 (<stdin>, line 58); backlink

    Unknown interpreted text role "class".

  • To specify columns to export and their order use :setting:`FEED_EXPORT_FIELDS`. Other feed exporters can also use this option, but it is important for CSV because unlike many other export formats CSV uses a fixed header.

    System Message: ERROR/3 (<stdin>, line 59); backlink

    Unknown interpreted text role "setting".

XML

Pickle

Marshal

Storages

When using the feed exports you define where to store the feed using one or multiple URIs (through the :setting:`FEEDS` setting). The feed exports supports multiple storage backend types which are defined by the URI scheme.

System Message: ERROR/3 (<stdin>, line 94); backlink

Unknown interpreted text role "setting".

The storages backends supported out of the box are:

Some storage backends may be unavailable if the required external libraries are not available. For example, the S3 backend is only available if the botocore library is installed.

Storage URI parameters

The storage URI can also contain parameters that get replaced when the feed is being created. These parameters are:

  • %(time)s - gets replaced by a timestamp when the feed is being created
  • %(name)s - gets replaced by the spider name

Any other named parameter gets replaced by the spider attribute of the same name. For example, %(site_id)s would get replaced by the spider.site_id attribute the moment the feed is being created.

Here are some examples to illustrate:

  • Store in FTP using one directory per spider:
    • ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json
  • Store in S3 using one directory per spider:
    • s3://mybucket/scraping/feeds/%(name)s/%(time)s.json

Storage backends

Local filesystem

The feeds are stored in the local filesystem.

  • URI scheme: file
  • Example URI: file:///tmp/export.csv
  • Required external libraries: none

Note that for the local filesystem storage (only) you can omit the scheme if you specify an absolute path like /tmp/export.csv. This only works on Unix systems though.

FTP

The feeds are stored in a FTP server.

  • URI scheme: ftp
  • Example URI: ftp://user:pass@ftp.example.com/path/to/export.csv
  • Required external libraries: none

FTP supports two different connection modes: active or passive. Scrapy uses the passive connection mode by default. To use the active connection mode instead, set the :setting:`FEED_STORAGE_FTP_ACTIVE` setting to True.

System Message: ERROR/3 (<stdin>, line 167); backlink

Unknown interpreted text role "setting".

S3

The feeds are stored on Amazon S3.

  • URI scheme: s3
  • Example URIs:
    • s3://mybucket/path/to/export.csv
    • s3://aws_key:aws_secret@mybucket/path/to/export.csv
  • Required external libraries: botocore

The AWS credentials can be passed as user/password in the URI, or they can be passed through the following settings:

You can also define a custom ACL for exported feeds using this setting:

Standard output

The feeds are written to the standard output of the Scrapy process.

  • URI scheme: stdout
  • Example URI: stdout:
  • Required external libraries: none

Settings

These are the settings used for configuring the feed exports:

System Message: ERROR/3 (<stdin>, line 224)

Unknown directive type "currentmodule".

.. currentmodule:: scrapy.extensions.feedexport

System Message: ERROR/3 (<stdin>, line 226)

Unknown directive type "setting".

.. setting:: FEEDS

FEEDS

System Message: ERROR/3 (<stdin>, line 231)

Unknown directive type "versionadded".

.. versionadded:: 2.1

Default: {}

A dictionary in which every key is a feed URI (or a :class:`pathlib.Path` object) and each value is a nested dictionary containing configuration parameters for the specific feed. This setting is required for enabling the feed export feature.

System Message: ERROR/3 (<stdin>, line 235); backlink

Unknown interpreted text role "class".

See :ref:`topics-feed-storage-backends` for supported URI schemes.

System Message: ERROR/3 (<stdin>, line 240); backlink

Unknown interpreted text role "ref".

For instance:

{
    'items.json': {
        'format': 'json',
        'encoding': 'utf8',
        'store_empty': False,
        'fields': None,
        'indent': 4,
    },
    '/home/user/documents/items.xml': {
        'format': 'xml',
        'fields': ['name', 'price'],
        'encoding': 'latin1',
        'indent': 8,
    },
    pathlib.Path('items.csv'): {
        'format': 'csv',
        'fields': ['price', 'name'],
    },
}

The following is a list of the accepted keys and the setting that is used as a fallback value if that key is not provided for a specific feed definition.

System Message: ERROR/3 (<stdin>, line 275)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_ENCODING

FEED_EXPORT_ENCODING

Default: None

The encoding to be used for the feed.

If unset or set to None (default) it uses UTF-8 for everything except JSON output, which uses safe numeric encoding (\uXXXX sequences) for historic reasons.

Use utf-8 if you want UTF-8 for JSON too.

System Message: ERROR/3 (<stdin>, line 289)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_FIELDS

FEED_EXPORT_FIELDS

Default: None

A list of fields to export, optional. Example: FEED_EXPORT_FIELDS = ["foo", "bar", "baz"].

Use FEED_EXPORT_FIELDS option to define fields to export and their order.

When FEED_EXPORT_FIELDS is empty or None (default), Scrapy uses the fields defined in :ref:`item objects <topics-items>` yielded by your spider.

System Message: ERROR/3 (<stdin>, line 301); backlink

Unknown interpreted text role "ref".

If an exporter requires a fixed set of fields (this is the case for :ref:`CSV <topics-feed-format-csv>` export format) and FEED_EXPORT_FIELDS is empty or None, then Scrapy tries to infer field names from the exported data - currently it uses field names from the first item.

System Message: ERROR/3 (<stdin>, line 304); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 309)

Unknown directive type "setting".

.. setting:: FEED_EXPORT_INDENT

FEED_EXPORT_INDENT

Default: 0

Amount of spaces used to indent the output on each level. If FEED_EXPORT_INDENT is a non-negative integer, then array elements and object members will be pretty-printed with that indent level. An indent level of 0 (the default), or negative, will put each item on a new line. None selects the most compact representation.

Currently implemented only by :class:`~scrapy.exporters.JsonItemExporter` and :class:`~scrapy.exporters.XmlItemExporter`, i.e. when you are exporting to .json or .xml.

System Message: ERROR/3 (<stdin>, line 321); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 321); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 325)

Unknown directive type "setting".

.. setting:: FEED_STORE_EMPTY

FEED_STORE_EMPTY

Default: False

Whether to export empty feeds (i.e. feeds with no items).

System Message: ERROR/3 (<stdin>, line 334)

Unknown directive type "setting".

.. setting:: FEED_STORAGES

FEED_STORAGES

Default: {}

A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes.

System Message: ERROR/3 (<stdin>, line 344)

Unknown directive type "setting".

.. setting:: FEED_STORAGE_FTP_ACTIVE

FEED_STORAGE_FTP_ACTIVE

Default: False

Whether to use the active connection mode when exporting feeds to an FTP server (True) or use the passive connection mode instead (False, default).

For information about FTP connection modes, see What is the difference between active and passive FTP?.

System Message: ERROR/3 (<stdin>, line 357)

Unknown directive type "setting".

.. setting:: FEED_STORAGE_S3_ACL

FEED_STORAGE_S3_ACL

Default: '' (empty string)

A string containing a custom ACL for feeds exported to Amazon S3 by your project.

For a complete list of available values, access the Canned ACL section on Amazon S3 docs.

System Message: ERROR/3 (<stdin>, line 368)

Unknown directive type "setting".

.. setting:: FEED_STORAGES_BASE

FEED_STORAGES_BASE

Default:

{
    '': 'scrapy.extensions.feedexport.FileFeedStorage',
    'file': 'scrapy.extensions.feedexport.FileFeedStorage',
    'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',
    's3': 'scrapy.extensions.feedexport.S3FeedStorage',
    'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
}

A dict containing the built-in feed storage backends supported by Scrapy. You can disable any of these backends by assigning None to their URI scheme in :setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend (without replacement), place this in your settings.py:

System Message: ERROR/3 (<stdin>, line 383); backlink

Unknown interpreted text role "setting".
FEED_STORAGES = {
    'ftp': None,
}

System Message: ERROR/3 (<stdin>, line 392)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS

FEED_EXPORTERS

Default: {}

A dict containing additional exporters supported by your project. The keys are serialization formats and the values are paths to :ref:`Item exporter <topics-exporters>` classes.

System Message: ERROR/3 (<stdin>, line 399); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 403)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS_BASE

FEED_EXPORTERS_BASE

Default:

{
    'json': 'scrapy.exporters.JsonItemExporter',
    'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
    'jl': 'scrapy.exporters.JsonLinesItemExporter',
    'csv': 'scrapy.exporters.CsvItemExporter',
    'xml': 'scrapy.exporters.XmlItemExporter',
    'marshal': 'scrapy.exporters.MarshalItemExporter',
    'pickle': 'scrapy.exporters.PickleItemExporter',
}

A dict containing the built-in feed exporters supported by Scrapy. You can disable any of these exporters by assigning None to their serialization format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter (without replacement), place this in your settings.py:

System Message: ERROR/3 (<stdin>, line 419); backlink

Unknown interpreted text role "setting".
FEED_EXPORTERS = {
    'csv': None,
}
</html>