Google Cloud Storage

The Google Cloud Storage Function enables you to collect and send data to Google Cloud Storage buckets. As a prerequisite to using this Function, you must have an active GCP service account, associated keys, and permissions. You can look up the Cloud Storage Quickstart guide for help with the setup.

To configure this function, the following credentials are required:

There are two functions associated with Google Cloud Storage:

1. Google Cloud Storage collector
2. Google Cloud Storage forwarder

Google Cloud Storage Collector

The Google Cloud Storage collector is used to read data from a selected Google Cloud Storage bucket. When the required credentials are entered, the service will access the cloud storage account and the contained data.

Google Credentials Tab

The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle ON Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.

File Information Tab

The data specified in this tab will indicate which folder the files will be read from. The users can indicate the specific Bucket name. Optionally, a Folder path can be specified if the data is not to be read from the root path.

There are three modes of data selection available under the File selection options, depending on the choice a different subset of files can be read by the function:

• All Files – This option will make the GCP Function load all files in the selected location.
• Specific File – This option is used to select a given file that is to be read by the Function. An input box called File Name will be shown where the selection is to be typed in. It is possible to use several files at once. 
• Use Regex – This option allows the users to use regular expressions to select a given type of file based on a custom scheme.

The users can also select an appropriate File format from a drop-down menu. The currently supported types are the following: CSV, Excel, JSON, and XML.

If the Excel format is chosen, an optional Include table header checkbox can be toggled. When applied a header for the structured data will be included. The delimiter by default is a comma (,), but that can be changed by typing in the desired character in the Delimiter field.

The After Collection tab allows for the users to choose if they want to delete the source file after the operation has finished. This is done by toggling the Remove option in the Source file action selection.

Google Cloud Storage Forwarder

The Google Cloud Storage forwarder enables the user to write/upload data to the configured Google Cloud Storage bucket.

Google Credentials Tab

The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle ON Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the Function will not run properly. Note that the characters are case-sensitive.

File Information Tab

Specify the Bucket to which the files are to be written. Optionally, a Folder (or subfolder) can be specified as a custom path, otherwise, the root directory will be used by default.

Output file information Tab

This section of the Google Cloud Storage forwarder is used to control the naming of the output files. There are two main choices in this regard:

• Collector Filename – The files will be written in accordance with the collector name from where they originate.
• Custom Filename – This will show an input box allowing for a custom file name to be set.

An optional parameter that can be enabled is the Append Timestamp setting which will apply a timestamp to each output file. The currently supported types are the following: CSV, Excel, JSON, and XML.

An optional Include table header checkbox indicates if the users want to include a header for the structured data. The delimiter by default is a comma (,), but that can be modified by typing in the desired character in the Delimiter field.

The Compress file option can be selected to compress the output files. When enabled, a drop-down menu is displayed for a format to be selected. Currently, only Zip and GZip formats are supported.