Google Cloud Storage

The Google Cloud Storage function enables you to collect and send data to Google Cloud Storage buckets. As a prerequisite to using this function, you must have an active GCP service account, associated keys, and permissions. You can look up the Cloud Storage Quickstart guide for help with the setup.

To configure this function, the following credentials are required:

Field	Description
Project ID	The unique string that is associated with your specific Google Cloud project. This is generated by you and must follow the requirements specified here. Note! Do not confuse Project ID with Project number which is a separate identifier. The project number is an automatically generated unique identifier for your project
Client E-mail	The e-mail address used by the GCP service.
Private Key	The JSON-generated key is used to login to the GCP service.
Bucket	The bucket in which your data is stored on the Google Cloud Storage.

There are two functions associated with Google Cloud Storage:

Google Cloud Storage Collector
Google Cloud Storage Forwarder

Google Cloud Storage Collector

The Google Cloud Storage collector is used to read data from a selected Google Cloud Storage bucket. When the required credentials are entered, the service will access the cloud storage account and the contained data.

Google credentials

The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in Secrets Wallet, you can toggle ON Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.

File information

The data specified in this tab will indicate which folder the files will be read from. The users can indicate the specific Bucket name. Optionally, a Folder path can be specified if the data is not to be read from the root path.

There are three modes of data selection available under the File selection options, depending on the choice a different subset of files can be read by the function:

All Files – This option will make the GCP function load all files in the selected location.
Specific File – This option is used to select a given file that is to be read by the function. An input box called File Name will be shown where the selection is to be typed in. It is possible to use several files at once.
Use Regex – This option allows the users to use regular expressions to select a given type of file based on a custom scheme.

You can also select an appropriate File format from a drop-down menu. The currently supported types are the following: CSV, Excel, JSON, and XML.

If the Excel format is chosen, an optional Include table header checkbox can be toggled. When applied a header for the structured data will be included. The delimiter by default is a comma (,), but that can be changed by typing in the desired character in the Delimiter field.

Note!

The Include table header and Delimiter options are applicable only for the CSV file format.

Note!

During collection,

Compressed files are automatically de-compressed.
The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.

For all supported archive file formats,

The archive must contain only a single file that is compressed.
The archive must not contain any directories.

The After Collection tab allows for you to choose if you want to delete the source file after the operation has finished. This is done by toggling the Remove option in the Source file action selection.

Google Cloud Storage Forwarder

The Google Cloud Storage forwarder enables the user to write/upload data to the configured Google Cloud Storage bucket.

Google Credentials

The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle on Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.

File Information

Specify the Bucket to which the files are to be written. Optionally, a Folder (or subfolder) can be specified as a custom path, otherwise, the root directory will be used by default.

Output file information

This section of the Google Cloud Storage forwarder is used to control the naming of the output files. There are two main choices in this regard:

Collector Filename – The files will be written in accordance with the collector name from where they originate. If the input file does not have a filename, the system generates a name based on the collector function, for example, Counter. The function appends the file format extension to the filename based on the file format selected below, if the selected extension is not in the filename.
Custom Filename – This will show an input box allowing for a custom file name to be set. The function appends the file format extension to the filename based on the file format selected below, so you do not need to add any extensions to the filename.

An optional parameter that can be enabled is the Append Timestamp setting which will apply a timestamp to each output file. The currently supported types are the following: CSV, Excel, JSON, and XML.

In File format, select the format of the output file(s) from the following options:

File format options	Description

File format options	Description
CSV	Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ','.
Excel	Select to send the output file in Excel format. Specify the name of the sheet in Sheet name.
JSON	Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record.
Buffer	Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder.

An optional Include table header checkbox indicates if the users want to include a header for the structured data. The delimiter by default is a comma (,), but that can be modified by typing in the desired character in the Delimiter field.

The Compress file option can be selected to compress the output files. When enabled, a drop-down menu is displayed for a format to be selected. Currently, only Zip and GZip formats are supported.