Kafka Batch Forwarding Agent (4.3)

The Kafka Batch Forwarding agent sends messages to Kafka but does not have any configuration settings. Because of this, it must be placed after an Analysis or Aggregation agent, which populates KafkaRecord UDRs with the messages.

The Kafka batch forwarding agent can send messages to multiple topics and partitions and transmits messages within Kafka transactions.

Consumers of these topics should typically set the Kafka property isolation.level to read_committed to ensure they only receive messages that were successfully committed at the end of a batch. For more details, see Kafka Documentation.

Workflow Example

A simple workflow with a Kafka batch forwarding agent can look like this:

The Kafka batch forwarding agent in a workflow

This workflow example has been created as follows:

Workflow Design

Create the workflow with the following agents:

Agent	Configuration
Disk	Collects files from disk.
Decoder	Decodes the input data to internal UDRs.
Analysis	Creates `KafkaRecord` UDRs and then maps the content of the input UDR. Sets the destination topic for the data.
Kafka	No configurations made in the agent.

Kafka Profile

The Kafka profile defines the broker to which the messages should be forwarded and you must have created it before you can select it in the Execution tab of the Workflow Properties.

Workflow Properties with Kafka profile selected

Analysis Agent

Configure the Analysis agent to create KafkaRecord UDRs, and then map the content of input UDR. Set the topic to which the data shall be forwarded.

kafka.KafkaRecord myUDR;

consume {
    
    myUDR = udrCreate(kafka.KafkaRecord);
    
    strToBA(myUDR.value, input.value);
    myUDR.topic = "myNewMZTopic";

    udrRoute(myUDR);

}