mirror of https://github.com/scrapy/scrapy.git
Adding documentation about Google Cloud Storage Feed Export
This commit is contained in:
parent
2bbbd02bda
commit
cb5f800b0f
|
|
@ -185,6 +185,27 @@ passed through the following settings:
|
|||
* :setting:`AWS_ACCESS_KEY_ID`
|
||||
* :setting:`AWS_SECRET_ACCESS_KEY`
|
||||
|
||||
.. _topics-feed-storage-gcs:
|
||||
|
||||
Google Cloud Storage (GCS)
|
||||
--------------------------
|
||||
|
||||
The feeds are stored on `Google Cloud Storage`_.
|
||||
|
||||
* URI scheme: ``gcs``
|
||||
* Example URIs:
|
||||
|
||||
* ``gcs://mybucket/path/to/export.csv``
|
||||
|
||||
* Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.
|
||||
|
||||
For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.
|
||||
|
||||
You can set a *Project ID* and *Access Control List (ACL)* through the following settings:
|
||||
|
||||
* :setting:`FEED_STORAGE_GCS_ACL`
|
||||
* :setting:`GCS_PROJECT_ID`
|
||||
|
||||
.. _topics-feed-storage-stdout:
|
||||
|
||||
Standard output
|
||||
|
|
@ -366,3 +387,4 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
|
|||
.. _Amazon S3: https://aws.amazon.com/s3/
|
||||
.. _boto: https://github.com/boto/boto
|
||||
.. _botocore: https://github.com/boto/botocore
|
||||
.. _Google Cloud Storage: https://cloud.google.com/storage/
|
||||
|
|
@ -749,6 +749,14 @@ The Feed Temp dir allows you to set a custom folder to save crawler
|
|||
temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
|
||||
:ref:`Amazon S3 <topics-feed-storage-s3>`.
|
||||
|
||||
.. setting:: FEED_STORAGE_GCS_ACL
|
||||
|
||||
FEED_STORAGE_GCS_ACL
|
||||
--------------------
|
||||
|
||||
The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
|
||||
For more information on how to set this value, please refer to `Google Cloud documentation <https://cloud.google.com/storage/docs/access-control/lists>`_.
|
||||
|
||||
.. setting:: FTP_PASSIVE_MODE
|
||||
|
||||
FTP_PASSIVE_MODE
|
||||
|
|
@ -786,6 +794,15 @@ Default: ``"anonymous"``
|
|||
The username to use for FTP connections when there is no ``"ftp_user"``
|
||||
in ``Request`` meta.
|
||||
|
||||
.. setting:: GCS_PROJECT_ID
|
||||
|
||||
GCS_PROJECT_ID
|
||||
-----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The Project ID that will be used when storing data on `Google Cloud Storage`_.
|
||||
|
||||
.. setting:: ITEM_PIPELINES
|
||||
|
||||
ITEM_PIPELINES
|
||||
|
|
@ -1371,3 +1388,4 @@ case to see how to enable and use them.
|
|||
.. _Amazon web services: https://aws.amazon.com/
|
||||
.. _breadth-first order: https://en.wikipedia.org/wiki/Breadth-first_search
|
||||
.. _depth-first order: https://en.wikipedia.org/wiki/Depth-first_search
|
||||
.. _Google Cloud Storage: https://cloud.google.com/storage/
|
||||
|
|
@ -168,6 +168,8 @@ FTP_USER = 'anonymous'
|
|||
FTP_PASSWORD = 'guest'
|
||||
FTP_PASSIVE_MODE = True
|
||||
|
||||
GCS_PROJECT_ID = None
|
||||
|
||||
HTTPCACHE_ENABLED = False
|
||||
HTTPCACHE_DIR = 'httpcache'
|
||||
HTTPCACHE_IGNORE_MISSING = False
|
||||
|
|
|
|||
Loading…
Reference in New Issue