Adding documentation about Google Cloud Storage Feed Export

This commit is contained in:
Júlio César Batista 2019-02-08 11:26:33 -02:00
parent 2bbbd02bda
commit cb5f800b0f
3 changed files with 42 additions and 0 deletions

View File

@ -185,6 +185,27 @@ passed through the following settings:
* :setting:`AWS_ACCESS_KEY_ID`
* :setting:`AWS_SECRET_ACCESS_KEY`
.. _topics-feed-storage-gcs:
Google Cloud Storage (GCS)
--------------------------
The feeds are stored on `Google Cloud Storage`_.
* URI scheme: ``gcs``
* Example URIs:
* ``gcs://mybucket/path/to/export.csv``
* Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.
For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.
You can set a *Project ID* and *Access Control List (ACL)* through the following settings:
* :setting:`FEED_STORAGE_GCS_ACL`
* :setting:`GCS_PROJECT_ID`
.. _topics-feed-storage-stdout:
Standard output
@ -366,3 +387,4 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
.. _Amazon S3: https://aws.amazon.com/s3/
.. _boto: https://github.com/boto/boto
.. _botocore: https://github.com/boto/botocore
.. _Google Cloud Storage: https://cloud.google.com/storage/

View File

@ -749,6 +749,14 @@ The Feed Temp dir allows you to set a custom folder to save crawler
temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
:ref:`Amazon S3 <topics-feed-storage-s3>`.
.. setting:: FEED_STORAGE_GCS_ACL
FEED_STORAGE_GCS_ACL
--------------------
The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
For more information on how to set this value, please refer to `Google Cloud documentation <https://cloud.google.com/storage/docs/access-control/lists>`_.
.. setting:: FTP_PASSIVE_MODE
FTP_PASSIVE_MODE
@ -786,6 +794,15 @@ Default: ``"anonymous"``
The username to use for FTP connections when there is no ``"ftp_user"``
in ``Request`` meta.
.. setting:: GCS_PROJECT_ID
GCS_PROJECT_ID
-----------------
Default: ``None``
The Project ID that will be used when storing data on `Google Cloud Storage`_.
.. setting:: ITEM_PIPELINES
ITEM_PIPELINES
@ -1371,3 +1388,4 @@ case to see how to enable and use them.
.. _Amazon web services: https://aws.amazon.com/
.. _breadth-first order: https://en.wikipedia.org/wiki/Breadth-first_search
.. _depth-first order: https://en.wikipedia.org/wiki/Depth-first_search
.. _Google Cloud Storage: https://cloud.google.com/storage/

View File

@ -168,6 +168,8 @@ FTP_USER = 'anonymous'
FTP_PASSWORD = 'guest'
FTP_PASSIVE_MODE = True
GCS_PROJECT_ID = None
HTTPCACHE_ENABLED = False
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_MISSING = False