The Google Cloud Storage Function enables you to collect and send data to Google Cloud Storage buckets. As a prerequisite to using this Function, you must have an active GCP service account, associated keys, and permissions. You can look up the Cloud Storage Quickstart guide for help with the setup.
To configure this function, the following credentials are required:
Field | Description |
Project ID | The unique string that is associated with your specific Google Cloud project. This is generated by the user and must follow the requirements specified here. Note! Do not confuse Project ID with Project number which is a separate identifier. The project number is an automatically generated unique identifier for your project |
Client E-mail | The e-mail address used by the GCP service. |
Private Key | The JSON-generated key is used to login to the GCP service. |
Bucket | The bucket in which your data is stored on the Google Cloud Storage. |
There are two functions associated with Google Cloud Storage:
- Google Cloud Storage collector
- Google Cloud Storage forwarder
Google Cloud Storage Collector
The Google Cloud Storage collector is used to read data from a selected Google Cloud Storage bucket. When the required credentials are entered, the service will access the cloud storage account and the contained data.
Google Credentials Tab
The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle ON Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.
File Information Tab
The data specified in this tab will indicate which folder the files will be read from. The users can indicate the specific Bucket name. Optionally, a Folder path can be specified if the data is not to be read from the root path.
There are three modes of data selection available under the File selection options, depending on the choice a different subset of files can be read by the function:
- All Files – This option will make the GCP Function load all files in the selected location.
- Specific File – This option is used to select a given file that is to be read by the Function. An input box called File Name will be shown where the selection is to be typed in. It is possible to use several files at once.
- Use Regex – This option allows the users to use regular expressions to select a given type of file based on a custom scheme.
The users can also select an appropriate File format from a drop-down menu. The currently supported types are the following: CSV, Excel, JSON, and XML.
If the Excel format is chosen, an optional Include table header checkbox can be toggled. When applied a header for the structured data will be included. The delimiter by default is a comma (,), but that can be changed by typing in the desired character in the Delimiter field.
Note!
The Include table header and Delimiter options are applicable only for the CSV file format.
Note!
During collection,
- Compressed files are automatically de-compressed.
- The type of archive file format is automatically identified based on the contents of the file instead of the file extension. The supported archive file formats are ZIP, gzip and zlib.
For all supported archive file formats,
- The archive must contain only a single file that is compressed.
- The archive must not contain any directories.
The After Collection tab allows for the users to choose if they want to delete the source file after the operation has finished. This is done by toggling the Remove option in the Source file action selection.
Google Cloud Storage Forwarder
The Google Cloud Storage forwarder enables the user to write/upload data to the configured Google Cloud Storage bucket.
Google Credentials Tab
The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle ON Secrets Wallet in configuration and select the appropriate credentials. If the data presented to the service is not valid, the Function will not run properly. Note that the characters are case-sensitive.
File Information Tab
Specify the Bucket to which the files are to be written. Optionally, a Folder (or subfolder) can be specified as a custom path, otherwise, the root directory will be used by default.
Output file information Tab
This section of the Google Cloud Storage forwarder is used to control the naming of the output files. There are two main choices in this regard:
- Collector Filename – The files will be written in accordance with the collector name from where they originate.
- Custom Filename – This will show an input box allowing for a custom file name to be set.
An optional parameter that can be enabled is the Append Timestamp setting which will apply a timestamp to each output file. The currently supported types are the following: CSV, Excel, JSON, and XML.
An optional Include table header checkbox indicates if the users want to include a header for the structured data. The delimiter by default is a comma (,), but that can be modified by typing in the desired character in the Delimiter field.
Note!
The Include table header and Delimiter options are applicable only for the CSV file format.
The Compress file option can be selected to compress the output files. When enabled, a drop-down menu is displayed for a format to be selected. Currently, only Zip and GZip formats are supported.