Versions Compared

Version Old Version 2 New Version 3
Changes made by

Casey Han

Casey Han

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The purpose of this collector is to enable messages collection from customer’s Kafka containers with ease. With Kafka Consumer, you collect messages from one or more topics from configured Kafka brokers, and process them in your stream. The collection method and batch size of the messages can also be configured. Once the messages are collected, they are converted to readable strings, and passed to the next function in the JSON format.

Insert excerpt
Kafka Producer
Kafka Producer
nameParameters and Configuration
nopaneltrue

Parameters

Under Configuration Parameters, you get Kafka brokers to set the Kafka broker to collect the messages from, Authentication send the record to. Authentication is to set the secret from Secrets Wallet with the credentials to collect from of the Kafka broker. You also get to fill in the Kafka topic, define how you will want the consumer to collect messages in Collection method and set the number of messages to collect per batch with Batch sizeBoth fields are mandatory.

Kafka brokers

Fill in the server host IP and the port number of the Kafka broker that you will want your stream to connect to.

Kafka broker.pngImage RemovedKafka broker.pngImage Added

Field

Description

Host

The hostname, IPV4 or IPV6 address of the Kafka broker.

Port

The listening port of the Kafka broker. The default port is 9092

You can add additional Kafka brokers to the Kafka Consumer by clicking on Add broker.

Authentication

Authentication.pngImage RemovedAuthentication.pngImage Added

Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Wallet on how to configure a secret using the Secrets Wallet.

If the Kafka broker does not have any authentication configured, select the None authentication type.

Note

Note!

SSL is enabled for all authentication types.

Configuration

Under Configuration, you get to fill in the Kafka topic

Kafka topic.pngImage Removed

, define how you will want the consumer to collect messages in Collection method and set the number of messages to collect per batch with Batch size.

Kafka topic

Kafka topic.pngImage Added

Key in the Topic name for the Kafka Consumer to retrieve the messages from. You may add more than one topic for the Kafka Consumer.

Note

Note!

Ensure to enter the full and correct topic name.

Collection method

Provides options on the messages you want to collect from the Kafka topic.

Collection method.pngImage RemovedCollection method.pngImage Added

Option

Description

All messages

With All messages selected, the function collects all messages in the Kafka topic, including the messages that has already been committed in the previous execution. A Consumer Group ID is arbitrarily assigned by Usage Engine.

New messages

With New messages, the function begins collecting from the last committed message of the Kafka topic. An alphanumeric Consumer Group ID is assigned by you. Only one Consumer Group ID is allowed.

The diagram below is an example of a Kafka Consumer collecting messages.

kafka_offset_diagram.png

Batch size

Batch size.pngImage RemovedBatch size.pngImage Added

Set the number of messages for each batch so that the stream commits to a certain number of messages per batch collection. If the topic contains 150 messages and the batch size is set to 100, the stream collects the first 100 messages from the topic to pass onto the subsequent operation. Then the next 50 messages are collected in the next batch, and the operation continues.

If there are messages less than the batch size, all the messages are collected. For example, if the topic has 50 messages, and the batch size is 100, all 50 messages are collected, and the stream continues with next operation.

The default value of the batch size is 100, which is also the minimum value. If you do not enter a value, then the default value will be used.