Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

The purpose of this collector is to enable messages collection from customer’s Kafka containers with ease. With Kafka Consumer, you collect messages from one or more topics from configured Kafka brokers, and process them in your stream. The collection method and batch size of the messages can also be configured. Once the messages are collected, they are converted to readable strings, and passed to the next function in the JSON format.

Configuration

Under Configuration, you get Kafka brokers to set the Kafka broker to collect the messages from, Authentication to set the secret with the credentials to collect from the Kafka broker. You also get to fill in the Kafka topic, define how you will want the consumer to collect messages in Collection method and set the number of messages to collect per batch with Batch size.

Kafka brokers

Fill in the server host IP and the port number of the Kafka broker that you will want your stream to connect to.

Kafka broker.png

Field

Description

Host

The hostname, IPV4 or IPV6 address of the Kafka broker.

Port

The listening port of the Kafka broker. The default port is 9092

You can add additional Kafka brokers to the Kafka Consumer by clicking on Add broker.

Authentication

Authentication.png

Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Wallet on how to configure a secret using the Secrets Wallet.

If the Kafka broker does not have any authentication configured, select the None authentication type.

Note!

SSL is enabled for all authentication types.

Kafka topic

Kafka topic.png

Key in the Topic name for the Kafka Consumer to retrieve the messages from. You may add more than one topic for the Kafka Consumer.

Note!

Ensure to enter the full and correct topic name.

Collection method

Provides options on the messages you want to collect from the Kafka topic.

Collection method.png

Option

Description

All messages

With All messages selected, the function collects all messages in the Kafka topic, including the messages that has already been committed in the previous execution. A Consumer Group ID is arbitrarily assigned by Usage Engine.

New messages

With New messages, the function begins collecting from the last committed message of the Kafka topic. An alphanumeric Consumer Group ID is assigned by you. Only one Consumer Group ID is allowed.

The diagram below is an example of a Kafka Consumer collecting messages.

kafka_offset_diagram.png

Batch size

Batch size.png

Set the number of messages for each batch so that the stream commits to a certain number of messages per batch collection. If the topic contains 150 messages and the batch size is set to 100, the stream collects the first 100 messages from the topic to pass onto the subsequent operation. Then the next 50 messages are collected in the next batch, and the operation continues.

If there are messages less than the batch size, all the messages are collected. For example, if the topic has 50 messages, and the batch size is 100, all 50 messages are collected, and the stream continues with next operation.

The default value of the batch size is 100, which is also the minimum value. If you do not enter a value, then the default value will be used.

  • No labels