Google Cloud Storage
- Former user (Deleted)
- Casey Han
- Former user (Deleted)
- Former user (Deleted)
The Google Cloud Storage function enables you to collect and send data to Google Cloud Storage buckets. As a prerequisite to using this function, you must have an active Google Cloud Platform (GCP) service account, associated keys, and permissions. You can look up the Cloud Storage Quickstart guide for help with the setup.
Disclaimer!
File storage maintenance, including storage, cleanup, and permissions, is your responsibility. As the function may involve transferral or removal of files in the Google Cloud Storage, it is recommended that you perform the necessary maintenance of your file storage before you run any streams with this function.
To configure this function, the following credentials are required:
Field | Description |
Project ID | The unique string that is associated with your specific Google Cloud project. This is generated by you and must follow the requirements specified here. Note! Do not confuse Project ID with Project number which is a separate identifier. The project number is an automatically generated unique identifier for your project |
Client email | The e-mail address used by the GCP service. |
Private key | The JSON-generated key is used to login to the GCP service. |
There are two functions associated with Google Cloud Storage:
Google Cloud Storage Collector
The Google Cloud Storage collector is used to read data from a selected Google Cloud Storage bucket. When the required credentials are entered, the service will access the cloud storage account and the contained data.
Google credentials
The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.
Alternatively, if you have stored the Google account credentials in Secrets wallet, you can toggle Secrets Wallet on in the configuration and select the appropriate Google secret.
File location
Specify the Bucket your data is stored on the Google Cloud Storage to which the files are to be collected from.
Optionally, a Folder path can be specified if the data is not to be read from the root path.
Note!
If the path to the folder is not specified, the root folder of the Google Cloud Storage bucket is selected by default.
After collection
Enable the After Collection toggle if you want to perform an action on the source file after the operation of the collector has completed. You can check one of the following actions below:
Action on source file | Action description |
|---|---|
Remove | Removes the source file after the operation of the collector is completed. |
Move to | Moves the source file to a destination folder. When this option is selected, you have to specify the folder path in the Folder field. The destination folder must be in the same bucket selected in File location. You can also check the Overwrite if file already exists check box, if you want to overwrite the file with the same name found in the folder. |
File information
There are three modes of data selection available under the File selection options, depending on the choice a different subset of files can be read by the function:
All files in folder – This option will make the GCP function load all files in the selected location.
Based on filename – This option is used to select a given file that is to be read by the function. An input box called File Name will be shown where the selection is to be typed in.
Based on regular expression – This option allows the users to use regular expressions to select a given type of file based on a custom scheme. An input box called File name regex will be shown where the expression is to be typed in.
You can also select an appropriate File format from a drop-down menu. The currently supported types are the following: CSV, Excel, JSON, and XML.
In the Delimiter field, you can select the delimiter. The delimiter by default is a comma (,).
If the Excel, or CSV format is chosen, an optional Include table header check box can be toggled. When applied, a header for the structured data will be included. For Excel, you can also select to include the table header in All sheets in file, or Specific sheet(s) and enter the specific sheet name in the Select sheet(s) field.
Google Cloud Storage Forwarder
The Google Cloud Storage forwarder enables the user to write/upload data to the configured Google Cloud Storage bucket.
Google Credentials
The following information is required to set up the connection to the Google Cloud Storage Bucket: Project ID, Client E-mail, and Private Key. If the data presented to the service is not valid, the function will not run properly. Note that the characters are case-sensitive.
Alternatively, if you have stored the Google account credentials in the Secrets Wallet, you can toggle on Secrets Wallet in configuration and select the appropriate credentials.
File Information
Optionally, a Folder (or subfolder) can be specified as a custom path, otherwise, the root directory will be used by default.
Output file information
This section of the Google Cloud Storage forwarder is used to control the naming of the output files. There are two main choices in this regard:
Collector Filename – The files will be written in accordance with the collector name from where they originate. If the input file does not have a filename, the system generates a name based on the collector function, for example, Counter. The function appends the file format extension to the filename based on the file format selected below, if the selected extension is not in the filename.
Custom Filename – This will show an input box allowing for a custom file name to be set. The function appends the file format extension to the filename based on the file format selected below, so you do not need to add any extensions to the filename.
An optional parameter that can be enabled is the Append Timestamp setting which will apply a timestamp to each output file. The currently supported types are the following: CSV, Excel, JSON, and XML.
In File format, select the format of the output file(s) from the following options:
File format options | Description |
|---|---|
CSV | Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). Specify a Delimiter for the CSV file format. The default value is ','. |
Excel | Select to send the output file in Excel format. Specify the name of the sheet in Sheet name. |
JSON | Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record. |
Buffer | Select to send the output file in Buffer format. If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the STFP Forwarder. |
An optional Include table header checkbox indicates if the users want to include a header for the structured data. The delimiter by default is a comma (,), but that can be modified by typing in the desired character in the Delimiter field.
The Compress file option can be selected to compress the output files. When enabled, a drop-down menu is displayed for a format to be selected. Currently, only Zip and GZip formats are supported.
Â