...
| Item | Description |
|---|---|
External References
| Select this menu item to enable the use of External References in the File System profile configuration. This can be used to configure the following fields: Amazon S3 file systems
HDFS file systems
For further information, see 8.10.4 Using External Reference in Agent Profile Fields and 8.10 External Reference Profile. |
...
When selecting Amazon S3 as file system, you will see two tabs; General and Advanced.
File System profile - Amazon S3 - General tab
...
In the Advanced tab, you can configure properties for the Amazon S3 File System client.
File System profile - Amazon S3 - Advanced tab
For information on how to configure the properties for Amazon S3 File System client, please refer to https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl.
HDFS
When selecting HDFS as file systems, you will see two tabs; General and Advanced.
File System profile - HDFS General tab
The General Tab
In the General tab you can find the following settings:
...
File System Type
...
Select a version of Hadoop from the drop-down box:
- Non HA - This version of Hadoop does not support high availability as it has only one NameNode.
- HA - This verison of Hadoop support high availability.
This setting only applies when you have selected Distributed File System as the File System Type.
...
Host
...
Enter the IP address or hostname of the NameNode in this field. See the Apache Hadoop Project documentation for further information about the NameNode.
...
Port
...
Enter the port number of the NameNode in this field.
The Advanced Tab
The Advanced tab contains Advanced Properties for the configuration of Kerberos authentication.
File System profile - HDFS Advanced tab
Kerberos is an authentication technology that uses a trusted third party to authenticate one service or user to another. Within Kerberos, this trusted third party is commonly referred to as the Key Distribution Center, or KDC. For HDFS, this means that the HDFS agent authenticates with the KDC using a user principal which must be pre-defined in the KDC. The HDFS cluster must be set up to use Kerberos, and the KDC must contain service principals for the HDFS NameNodes. For information on how to set up a HDFS cluster with Kerberos, see the Hadoop Users Guide at http://www.hadoop.apache.org.
In order to perform authentication towards the KDC without a password, the HDFS agent requires a keytab file.
You can set the advanced properties in the Advanced Properties dialog to activate and configure Kerberos authentication.
The following advanced properties are related to Kerberos authentication. Refer to the Advanced Properties dialog for examples.
...
Set the value to kerberos to activate Kerberos authentication.
| Note | ||
|---|---|---|
| ||
Due to limitations in the Apache Hadoop client libraries, if you change this property, you may be required to restart the ECs where workflows containing the HDFS agent is going to run. |
...
The following properties are also included in the Advanced tab, but only apply if you have selected the HA version of Hadoop in the General tab:
...
| Note | ||
|---|---|---|
| ||
If you are using Kerberos authentication, it is recommended that you only run the HDFS agents toward one HDFS cluster per EC. This is because the Kerberos client library of HDFS relies on static properties and configurations that are global for the whole JVM. This means that one workflow running the HDFS agents could impact another workflow running the HDFS agents within the same EC process. Due to this limitation, you must also restart the EC for some configuration changes to the Advanced Properties. |
...
Create a properties file containing the advanced configurations.
...
| title | Example - Properties file with advanced configurations |
|---|
| Code Block | ||||
|---|---|---|---|---|
| ||||
ADV_PROP=hadoop.security.authentication\=kerberos\n\
java.security.krb5.kdc\=kdc.example.com\n\
dr.kerberos.client.principal\=mzadmin@EXAMPLE.COM\n\
dr.kerberos.client.keytabfile\=/home/mzadmin/keytabs/ex.keytab |
| Note | ||
|---|---|---|
| ||
All "=" characters need to be escaped. |
...
.