Versions Compared

Version Old Version 3 New Version Current
Changes made by

Casey Han

Jenni Wallin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The purpose of this collector is to enable messages message collection from customer’s Kafka containers with ease. With Kafka Consumerconsumer, you collect messages from one or more topics from configured Kafka brokers, and process them in your stream. The collection method and batch size of the messages can also be configured. Once the messages are collected, they are converted to readable strings, and passed to the next function in the JSON format. Insert excerptKafka ProducerKafka ProducernameParameters and Configurationnopaneltrue

Parameters

Under Parameters, you get Kafka You can configure this function in the following settings. All these configurations are mandatory.

send the record to.

  • collect the message(s) from.

  • Authentication is to set the secret from Secrets Wallet with the credentials of the Kafka broker

. Both fields are mandatory. Kafka

  • .

  • Kafka topicto set the topic to collect the messages from.

  • Collection methoddefines how you will want the consumer to collect the messages.

  • Batch size sets the number of messages to collect per batch.

Kafka brokers
Anchor
Kafka-brokers
Kafka-brokers

Fill in the server host IP and the port number of the Kafka broker that you will want your stream to connect to.

Kafka broker.pngImage Removed

collect the messages from.

Field

Description

Host

The hostname, IPV4 IPv4 or IPV6 IPv6 address of the Kafka broker.

Port

The listening port of the Kafka broker. The default port is 9092

Kafka broker.pngImage Added

You can add additional Kafka brokers to the Kafka Consumer by clicking on Add broker.

Authentication
Anchor
Authentication

Authentication.pngImage Removed

Authentication

Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Walletwallet on how to configure a secret using the Secrets Wallet.

Authentication.pngImage Added

If the Kafka broker does not have any authentication configured, select the None authentication type.

note

you may disable Authentication by toggling it off.

Note!

SSL is enabled for all authentication types.

Kafka topic

Configuration

Under Configuration, you get to fill in the Kafka topic, define how you will want the consumer to collect messages in Collection method and set the number of messages to collect per batch with Batch size.

Kafka topic

Kafka topic.pngImage Removed

Anchor
Kafka-topic
Kafka-topic

Key in the Topic name for the Kafka Consumer to retrieve collect the messages from. You may add more than one topic for the Kafka Consumer.

note

Note!

Ensure to enter the full and correct topic name.

Kafka topic.pngImage Added

Collection method
Anchor
Collection-method
Collection-method

Provides options on the messages you want method to collect messages from the Kafka topic(s).

Collection method.pngImage Removed

Option

Description

All messages

With All messages selected, the function collects all messages in the Kafka topic, including the messages that has already been committed in the previous execution. A Consumer Group ID is arbitrarily assigned by Usage Engine.

New messages

With New messages, the function begins collecting from the last committed message of the Kafka topic. An alphanumeric Consumer Group ID is assigned by you. Only one Consumer Group ID is allowed.

Collection method.pngImage Added

The diagram below is an example of a how Kafka Consumer collecting collects messages using either methods.

kafka_offset_diagram.pngBatch size.pngImage Removed

Batch size

Anchor
Batch-size
Batch-size

Set the number of messages for each batch so that the stream commits to a certain number of messages per batch collection. If the topic contains 150 messages and the batch size is set to 100, the stream collects the first 100 messages from the topic to pass onto the subsequent operation. Then the next 50 messages are collected in the next batch, and the operation continues.

Batch size.pngImage Added

If there are messages less than the batch size, all the messages are collected. For example, if the topic has 50 messages, and the batch size is 100, all 50 messages are collected, and the stream continues with next operation.

The default value of the batch size is 100, which is also the minimum value. If you do not enter a value, then the default value will be used.