/
File System (3.1)

File System (3.1)

The File System Profile is used for making file system specific configurations, currently used by:

  • Amazon S3 collection agent
  • Amazon S3 forwarding agent
  • GCP Storage collection agent
  • GCP Storage forwarding agent
  • HDFS collection agent
  • HDFS forwarding agent
  • System Importer
  • System Exporter

Configuration

The configurations will vary depending on the selected file system, and each file system will be described separately below.

Menus

The External Reference button is specific for the File System profile configurations.

ItemDescription

External References


Select this menu item to enable the use of External References in the File System profile configuration. This can be used to configure the following fields:

Amazon S3 file systems

  • Access Key
  • Secret Key
  • Bucket
  • Region
  • Advanced Properties

GCP Storage file systems

  • Use JSON File
    • Credentials File
    • Bucket
  • Form
    • Project Id
    • Private Key Id
    • Private Key
    • Client Email
    • Client Id
    • Other Information
    • Bucket

HDFS file systems

  • Host
  • Port
  • Advanced Properties
  • Replication


Amazon S3

When selecting Amazon S3 as file system, you will see two tabs; General and Advanced.

General Tab

The following settings are available in the General tab in the Amazon S3 File System profile:

SettingDescription

File System Type

Select which file system type this profile should be applied for. You can choose either Amazon S3 or HDFS.
Credentials from EnvironmentSelect this check box in order to pick up the credentials from the environment instead of entering them in this profile. If this checkbox is selected, the Access Key and Secret Key fields will be disabled.

Access Key


Enter the access key for the user who owns the Amazon S3 account in this field.

If you want to set a parameter, select the Parameterized checkbox and enter the parameter name using ${} syntax, see Profiles(3.1) for more information on how parameterization works (in this mode the regular access key field is disabled).

Secret Key


Enter the secret key for the stated access key in this field.

If you want to set a parameter, select the Parameterized checkbox and enter the parameter name using ${} syntax, see Profiles(3.1) for more information on how parameterization works (in this mode the regular secret key field is disabled).

Region from EnvironmentSelect this check box in order to pick up the region from the environment instead of entering the region in this profile. If this check box is selected, the Region field will be disabled.
RegionEnter the name of the Amazon S3 region in this field.

Bucket

Enter the name of the Amazon S3 bucket in this field.

Advanced Tab

In the Advanced tab, you can configure properties for the Amazon S3 File System client. 

For information on how to configure the properties for Amazon S3 File System client, please refer to https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl.

GCP Storage

When selecting GCP Storage as file system, you will see the tab General.

GCP Storage File System - General Tab

Json File

The following settings are available when you have selected Use Json File as the Input Option in the GCP Profile.