Kafka Consumer

The purpose of this collector is to enable message collection from Kafka containers with ease. With Kafka Consumer, you collect messages from one or more topics from configured Kafka brokers, and process them in your stream. The collection method and batch size of the messages can also be configured. Once the messages are collected, they are converted to readable strings, and passed to the next function in the JSON format.

You can configure this function in the following settings. All these configurations are mandatory.

Kafka brokers to set the Kafka broker to collect the message(s) from.
Authentication is to set the secret from Secrets Wallet with the credentials of the Kafka broker.
Kafka topic to set the topic to collect the messages from.
Collection method defines how you will want the consumer to collect the messages.
Batch size sets the number of messages to collect per batch.

Kafka brokers

Fill in the server host IP and the port number of the Kafka broker that you will want your stream to collect the messages from.

Field	Description

Field	Description
Host	The hostname, IPv4 or IPv6 address of the Kafka broker.
Port	The listening port of the Kafka broker. The default port is 9092

You can add additional Kafka brokers by clicking on Add broker.

Authentication

Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Wallet on how to configure a secret using the Secrets Wallet.

If the Kafka broker does not have any authentication configured, you may disable Authentication by toggling it off.

Note!

SSL is enabled for all authentication types.

Kafka topic

Key in the Topic name for the Kafka Consumer to collect the messages from. You may add more than one topic.

Note!

Ensure to enter the full and correct topic name.

Collection method

Provides options on the method to collect messages from the Kafka topic(s).

Option	Description

Option	Description
All messages	With All messages selected, the function collects all messages in the Kafka topic, including the messages that has already been committed in the previous execution. A Consumer Group ID is arbitrarily assigned by Usage Engine.
New messages	With New messages, the function begins collecting from the last committed message of the Kafka topic. An alphanumeric Consumer Group ID is assigned by you. Only one Consumer Group ID is allowed.

The diagram below is an example of how Kafka Consumer collects messages using either methods.

Batch size

Set the number of messages for each batch so that the stream commits to a certain number of messages per batch collection. If the topic contains 150 messages and the batch size is set to 100, the stream collects the first 100 messages from the topic to pass onto the subsequent operation. Then the next 50 messages are collected in the next batch, and the operation continues.

If there are messages less than the batch size, all the messages are collected. For example, if the topic has 50 messages, and the batch size is 100, all 50 messages are collected, and the stream continues with next operation.

The default value of the batch size is 100, which is also the minimum value. If you do not enter a value, then the default value will be used.