HDFS Forwarding Agent(3.0)

The HDFS forwarding agent creates files on the remote file system containing the received data. Files are created when a Begin Batch message is received, and closed when an End Batch message is received. In addition, theĀ Filename TemplateĀ service offers the possibility to compress (gzip) the files, or to further process them, using commands.

To ensure that downstream systems will not use the files until they are closed, they are stored in a temporary directory until the End Batch message is received. This behavior also applies to Cancel Batch messages. If a Cancel Batch is received, file creation is cancelled.


The chapter contains the following sections: