External References

Select this menu item to enable External References in the File System profile configuration. This can be used to configure the following fields:

Amazon S3 file systems

• Access Key

• Secret Key

• Bucket

• Region

• Advanced Properties

GCP Storage file systems

• Use JSON File

  • Credentials File

  • Bucket

• Form

  • Project Id

  • Private Key Id

  • Private Key

  • Client Email

  • Client Id

  • Other Information

  • Bucket

HDFS file systems

• Host

• Port

• Advanced Properties

• Replication

Repository URL

The URL to the repository. 

Token

Token to access the repository. This field is optional.

Use Secrets Profile

Select the checkbox to use a Secrets Profile to get the Token.

Get Branches

Click this button to fetch the branches from the repository. If the connection is working the Branch combo box will be populated. If the connection fails, an error dialog will be shown.

Branch

Select the branch to use. 

Preview Repository

Click here to browse the folders in the repository. It is only possible when the configuration is saved.

File System Type

Select which file system type this profile should be applied for. You can choose either Amazon S3 or HDFS.

Credentials from Environment

Select this check box to pick up the credentials from the environment instead of entering them in this profile. If this checkbox is selected, the Access Key and Secret Key fields will be disabled.

Access Key

Enter the access key for the user who owns the Amazon S3 account in this field.

If you want to set a parameter, select the Parameterized checkbox and enter the parameter name using ${} syntax, see Profiles(4.0) for more information on how parameterization works (in this mode the regular access key field is disabled).

Secret Key

Enter the secret key for the stated access key in this field.

If you want to set a parameter, select the Parameterized checkbox and enter the parameter name using ${} syntax, see Profiles(4.0) for more information on how parameterization works (in this mode the regular secret key field is disabled).

Region from Environment

Select this check box to pick up the region from the environment instead of entering the region in this profile. If this check box is selected, the Region field will be disabled.

Region

Enter the name of the Amazon S3 region in this field.

Bucket

Enter the name of the Amazon S3 bucket in this field.

Bucket

Enter the name of the GCP Storage bucket in this field.

Use GCP Profile

Select the checkbox and then choose an existing GCP Profile if the Authentication Details should be derived from a GCP Profile instead of adding them directly to this profile.

File System Type

Select which file system type this profile should be applied for. You can choose either Amazon S3 or HDFS.

Hadoop Mode

Select the type of Hadoop from the drop-down box:

• Non HA - This version of Hadoop does not support high availability as it has only one NameNode.

• HA - This version of Hadoop supports high availability.

Host

Enter the IP address or hostname of the NameNode in this field. See the Apache Hadoop Project documentation for further information about the NameNode.

Port

Enter the port number of the NameNode in this field.

Replication

Enter the number for HDFS to configure the replication factor. Replication is used for fault tolerance and more information regarding replication be found at: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication

hadoop.security.authentication

Set the value to kerberos to activate Kerberos authentication.

dfs.namenode.kerberos.principal

This sets the service principal to use for the HDFS NameNode. This must be predefined in the KDC. The service principal is expected to be in the form of nn/<host>@<REALM> where <host> is the host where the service is running and <REALM> is the name (in uppercase) of the Kerberos realm.

java.security.krb5.kdc

This specifies the hostname of the Key Distribution Center.

java.security.krb5.realm

This sets the name of the Kerberos realm. Uppercase only.

dr.kerberos.client.keytabfile

This sets the keytab file to use for authentication. A keytab must be predefined using Kerberos tools. The keytab must be generated for the user principal in dr.kerberos.client.principal. This filepath must be on a file system that can be reached from the EC process. The user that launches the EC must also have read permissions for this file.

dr.kerberos.client.principal

This sets the user principal that the HDFS agent authenticates as. This must be predefined in the KDC. User principals are expected to be in the form of <user>@<REALM> where <user> is typically a username and <REALM> is the name (in uppercase) of the Kerberos realm.

sun.security.krb5.debug

Set this value to true to activate debug output for Kerberos.

fs.defaultFS

This sets the HDFS filesystem path prefix.

dfs.nameservices

This sets the logical name for the name services.

dfs.ha.namenodes.<nameservice ID>

This sets the unique identifiers for each NameNode in the name service.

dfs.namenode.rpc-address.<nameservice ID>.<name node ID>

This sets the fully-qualified RPC address for each NameNode to listen on.

dfs.namenode.http-address.<nameservice ID>.<name node ID>

This sets the fully-qualified HTTP address for each NameNode to listen on.

dfs.client.failover.proxy.provider.<nameservice ID>

This sets the Java class that HDFS clients use to contact the Active NameNode.