Kafka Batch Forwarding Agent

The Kafka Batch Forwarding agent sends messages to Kafka but does not have any configuration settings. Because of this, it must be placed after an Analysis or Aggregation agent, which populates KafkaRecord UDRs with the messages.

The Kafka batch forwarding agent can send messages to multiple topics and partitions and transmit messages within Kafka transactions.

Consumers of these topics should typically set the Kafka property isolation.level to read_committed to ensure they only receive messages that were successfully committed at the end of a batch. For more details, see Kafka Documentation.

Workflow Example

A simple workflow with a Kafka batch forwarding agent:

The Kafka batch forwarding agent in a workflow

Workflow Design

Create the workflow with the following agents:

Agent	Configuration
Disk	Collects files from disk.
Decoder	Decodes the input data to internal UDRs.
Analysis	Creates `KafkaRecord` UDRs and then maps the content of the input UDR. Sets the destination topic for the data.
Kafka	No configurations made in the agent.

Kafka Profile

The Kafka profile defines the broker to which the messages should be forwarded. You need to create the Kafka profile before selecting it in the Execution tab of the Workflow Properties.

Workflow Properties with Kafka profile selected

Analysis Agent

Configure the Analysis agent to create KafkaRecord UDRs, and then map the content of the input UDR. Set the topic where the data should be forwarded.

kafka.KafkaRecord myUDR;

consume {
    
    myUDR = udrCreate(kafka.KafkaRecord);
    
    strToBA(myUDR.value, input.value);
    myUDR.topic = "myNewMZTopic";

    udrRoute(myUDR);

}