9.39.3 HDFS Forwarding Agent

The HDFS forwarding agent creates files on the remote file system containing the received data. Files are created when a Begin Batch message is received, and closed when an End Batch message is received. In addition, the Filename Template service offers the possibility to compress (gzip) the files, or to further process them, using commands.

To ensure that downstream systems will not use the files until they are closed, they are stored in a temporary directory until the End Batch message is received. This behavior also applies to Cancel Batch messages. If a Cancel Batch is received, file creation is cancelled.