RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/feed-exports.rst

7.0 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Feed exports

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "versionadded".

.. versionadded:: 0.10

One of the most frequently required features when implementing scrapers is being able to store the scraped data properly and, quite often, that means generating an "export file" with the scraped data (commonly called "export feed") to be consumed by other systems.

Scrapy provides this functionality out of the box with the Feed Exports, which allows you to generate a feed with the scraped items, using multiple serialization formats and storage backends.

Serialization formats

For serializing the scraped data, the feed exports use the :ref:`Item exporters <topics-exporters>`. These formats are supported out of the box:

System Message: ERROR/3 (<stdin>, line 23); backlink

Unknown interpreted text role "ref".

But you can also extend the supported format through the :setting:`FEED_EXPORTERS` setting.

System Message: ERROR/3 (<stdin>, line 31); backlink

Unknown interpreted text role "setting".

JSON

JSON lines

CSV

XML

Pickle

Marshal

Storages

When using the feed exports you define where to store the feed using a URI (through the :setting:`FEED_URI` setting). The feed exports supports multiple storage backend types which are defined by the URI scheme.

System Message: ERROR/3 (<stdin>, line 89); backlink

Unknown interpreted text role "setting".

The storages backends supported out of the box are:

Some storage backends may be unavailable if the required external libraries are not available. For example, the S3 backend is only available if the boto library is installed.

Storage URI parameters

The storage URI can also contain parameters that get replaced when the feed is being created. These parameters are:

  • %(time)s - gets replaced by a timestamp when the feed is being created
  • %(name)s - gets replaced by the spider name

Any other named parameter gets replaced by the spider attribute of the same name. For example, %(site_id)s would get replaced by the spider.site_id attribute the moment the feed is being created.

Here are some examples to illustrate:

  • Store in FTP using one directory per spider:
    • ftp://user:password@ftp.example.com/scraping/feeds/%(name)s/%(time)s.json
  • Store in S3 using one directory per spider:
    • s3://mybucket/scraping/feeds/%(name)s/%(time)s.json

Storage backends

Local filesystem

The feeds are stored in the local filesystem.

  • URI scheme: file
  • Example URI: file:///tmp/export.csv
  • Required external libraries: none

Note that for the local filesystem storage (only) you can omit the scheme if you specify an absolute path like /tmp/export.csv. This only works on Unix systems though.

FTP

The feeds are stored in a FTP server.

  • URI scheme: ftp
  • Example URI: ftp://user:pass@ftp.example.com/path/to/export.csv
  • Required external libraries: none

S3

The feeds are stored on Amazon S3.

  • URI scheme: s3
  • Example URIs:
    • s3://mybucket/path/to/export.csv
    • s3://aws_key:aws_secret@mybucket/path/to/export.csv
  • Required external libraries: boto

The AWS credentials can be passed as user/password in the URI, or they can be passed through the following settings:

Standard output

The feeds are written to the standard output of the Scrapy process.

  • URI scheme: stdout
  • Example URI: stdout:
  • Required external libraries: none

Settings

These are the settings used for configuring the feed exports:

System Message: ERROR/3 (<stdin>, line 206)

Unknown directive type "currentmodule".

.. currentmodule:: scrapy.contrib.feedexport

System Message: ERROR/3 (<stdin>, line 208)

Unknown directive type "setting".

.. setting:: FEED_URI

FEED_URI

Default: None

The URI of the export feed. See :ref:`topics-feed-storage-backends` for supported URI schemes.

System Message: ERROR/3 (<stdin>, line 215); backlink

Unknown interpreted text role "ref".

This setting is required for enabling the feed exports.

System Message: ERROR/3 (<stdin>, line 220)

Unknown directive type "setting".

.. setting:: FEED_FORMAT

FEED_FORMAT

The serialization format to be used for the feed. See :ref:`topics-feed-format` for possible values.

System Message: ERROR/3 (<stdin>, line 225); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 228)

Unknown directive type "setting".

.. setting:: FEED_STORE_EMPTY

FEED_STORE_EMPTY

Default: False

Whether to export empty feeds (ie. feeds with no items).

System Message: ERROR/3 (<stdin>, line 237)

Unknown directive type "setting".

.. setting:: FEED_STORAGES

FEED_STORAGES

Default:: {}

A dict containing additional feed storage backends supported by your project. The keys are URI schemes and the values are paths to storage classes.

System Message: ERROR/3 (<stdin>, line 247)

Unknown directive type "setting".

.. setting:: FEED_STORAGES_BASE

FEED_STORAGES_BASE

Default:

{
    '': 'scrapy.contrib.feedexport.FileFeedStorage',
    'file': 'scrapy.contrib.feedexport.FileFeedStorage',
    'stdout': 'scrapy.contrib.feedexport.StdoutFeedStorage',
    's3': 'scrapy.contrib.feedexport.S3FeedStorage',
    'ftp': 'scrapy.contrib.feedexport.FTPFeedStorage',
}

A dict containing the built-in feed storage backends supported by Scrapy.

System Message: ERROR/3 (<stdin>, line 264)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS

FEED_EXPORTERS

Default:: {}

A dict containing additional exporters supported by your project. The keys are URI schemes and the values are paths to :ref:`Item exporter <topics-exporters>` classes.

System Message: ERROR/3 (<stdin>, line 271); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 275)

Unknown directive type "setting".

.. setting:: FEED_EXPORTERS_BASE

FEED_EXPORTERS_BASE

Default:

FEED_EXPORTERS_BASE = {
    'json': 'scrapy.contrib.exporter.JsonItemExporter',
    'jsonlines': 'scrapy.contrib.exporter.JsonLinesItemExporter',
    'csv': 'scrapy.contrib.exporter.CsvItemExporter',
    'xml': 'scrapy.contrib.exporter.XmlItemExporter',
    'marshal': 'scrapy.contrib.exporter.MarshalItemExporter',
}

A dict containing the built-in feed exporters supported by Scrapy.

</html>