HDFS Forwarding Agent Configuration(3.1)

Owned by Jenni Wallin
May 10, 2023
2 min read

HDFS Tab

ItemDescription
ProfileSelect the File System profile you want the agent to use, see File System Profile - HDFS (3.1) for further information about this profile.

Input Type

The agent can act on two input types: bytearray or MultiForwardingUDR. The default input type is bytearray. If nothing else is stated, the documentation refers to input of type bytearray.

For more information, see MultiForwardingUDR(3.0).

For an example see  HDFS Forwarding Agent MultiForwardingUDR Example(3.1).

Directory

Absolute path to the target directory on the remote file system, where the forwarded files will be stored.

The files will be temporarily stored in the automatically created subdirectory DR_TMP_DIR, in the target directory. When an End Batch message is received, the files are moved from the subdirectory to the target directory.

Create Directory

Select this check box to create the directory, or the directory structure, of the path that you specify in Directory.

Note

The directories are created when the workflow is executed.

Compression

Compression type of the target files. Determines if the agent will compress the files or not.

  • No Compression - the agent does not compress the files. Default setting.

  • Gzip - the agent compresses the files using gzip.

    Note

    No extra extension will be appended to the target filenames, even if compression is selected. The configuration of the filenames is managed in the Filename Template tab only.

Command

If a Command is supplied, it will be executed on each successfully closed temporary file, using the parameter values declared in Arguments.

Note

At this point the temporary file is created and closed, however the final filename has not yet been created.

The entered Command has to exist in the execution environment, either including an absolute path, or to be found in the PATH for the execution environment.

Arguments

This field is optional. Each entered parameter value has to be separated from the preceding value with a space.

The temporary filename is inserted as the second last parameter, and the final filename is inserted as the last parameter, automatically. This means that if, for instance, no parameter is given in the field, the arguments will be as follows:

$1=<temporary_filename> $2=<final_filename>

If three parameters are given in the field Arguments, the arguments are set as:

$1=<parameter_value_#1>
$2=<parameter_value_#2>
$3=<parameter_value_#3>
$4=<temporary_filename>
$5=<final_filename>

Produce Empty Files

If enabled, files will be produced although containing no data.

Filename Template Tab

The names of the created files are determined by the settings in the Filename Template tab. For a detailed description of the Filename Template tab, see Workflow Template (3.0).