Kafka Real-Time Collection Agent

The Kafka real-time collection agent consumes messages from Kafka. Several workflows can collect messages from the same set of topics in parallel but from different partitions. Starting and stopping workflows will automatically rebalance which partitions the workflows collect from, see Automatic Scale Out and Rebalancing.

Workflow Example

A simple workflow with a Kafka real-time collection agent can look like this:

Workflow with Kafka real-time collection agent

This workflow example has been created as follows:

Workflow Design

Create the workflow with the following agents:

Agent	Configuration
Kafka	Collects messages from Kafka.
Analysis	Receives `KafkaRecord` UDRs and creates output UDRs based on the contents of the input UDR. The offset is used to create a unique id.
Encoder	Encodes the data to the format the files will be forwarded in.
Disk	Creates files with the defined format and stores them in the stated directory.

Kafka Collector

Configure the Kafka real-time collector agent to minimize potential data loss by setting Offset management to At Least Once.

Kafka Collection Agent.

Analysis Agent

Configure the Analysis agent to create an output UDR and then map the content of the Kafka message. A unique id is created by using the offset from the input UDR.

consume {

    Default.UFL_test.test_TI myUDR = udrCreate(Default.UFL_test.test_TI);
    myUDR.offSet = (string)input.offset;
    myUDR.value = baToStr(input.value);
    
    udrRoute(myUDR);

}