Kafka Consumer

Note!

Before you configure the Kafka Consumer function:

Ensure that the Kafka cluster is reachable from UsageCloud.
Configure the Kafka ACLs so that the client identity has the required TOPIC and GROUP permissions.

For more information, see Prerequisites for using Kafka functions.

The purpose of the Kafka Consumer collector is to enable easy message collection from Kafka brokers. You collect messages from one or more topics from configured Kafka brokers and process them in your stream. You can also configure the collection method and batch size of the messages. Once the messages are collected, they are converted to readable strings and passed to the next function in JSON format.

Page navigation:

1 Kafka brokers
2 Authentication
3 Kafka topic
4 Collection method
5 Batch size

screencapture-neslo-release-2-547-tentative-kube-dazzl-io-streams-1dc33ae6-c261-400d-9193-6ef5925bc362-2026-06-10-11_11_38.png

Kafka Consumer collector configuration

Setting	Description	Field requirement

Setting	Description	Field requirement
Kafka brokers Enter the host IP address and port number of the Kafka broker you want your stream to collect messages from.
Host	The hostname, IPv4 or IPv6 address of the Kafka broker.	Required
Port	The listening port of the Kafka broker. The default port is 9092	Required
Add broker	Click Add broker to add additional Kafka brokers.	Optional
Authentication
Secrets wallet	Note! This option only appears when the Authentication toggle is enabled. Select the Secrets wallet radio button to select a secret with the Kafka broker secret type. For more details, see Secrets wallet.	Optional
Direct input	Note! This option only appears when the Authentication toggle is enabled. Direct input authentication options for PLAIN/SCRAM Select the Direct input radio button to choose between two authentication types: PLAIN/SCRAM SASL mechanism - Select the SASL authentication mechanism that the Kafka broker is configured to use. Username - The username used to authenticate the Kafka client with the broker. Password - The password associated with the username used to authenticate the Kafka client with the broker. Note! The Password field only accepts a parameter reference and you need to use Global parameters. Direct input authentication options for OAuth OAuth Host - The hostname or IP address of the OAuth server that issues access tokens. Path - The HTTP path on the OAuth server used to request access tokens. Client ID - The OAuth client identifier that represents this Kafka client application when requesting tokens from the OAuth server. Client Secret - The secret associated with the OAuth client ID, used by the Kafka client to authenticate with the OAuth server when requesting tokens. Note! The Client Secret field only accepts a parameter reference and you have to use Global parameters.	Optional
None	If the Kafka broker does not require authentication, you can disable Authentication by toggling it off. Note! SSL is enabled for all authentication types.	Optional
Kafka topic
Topic name	Enter the Topic name for the Kafka Consumer to collect the messages from. You can add more than one topic.	Required
Collection method Determines which messages you want to collect from the Kafka topics.
Collect	All messages - The function collects all messages in the Kafka topic, including the messages that have already been committed in the previous execution. A Consumer group ID is arbitrarily assigned by UsageCloud. New messages - The function begins collecting from the last committed message of the Kafka topic. An alphanumeric Consumer group ID is assigned by you. Only one Consumer group ID is allowed. From offset - The function collects messages starting from a specified offset in a specified partition. With this option, only one topic can be selected. Use this option to compensate for a previous stream failure or rollback. The offset value is obtained from the Kafka broker. Once collection has caught up to the latest message, switch back to New messages using the same Consumer group ID to resume normal collection.	Required
Consumer group ID	Note! This setting appears only when you select New message. A unique identifier that represents this Kafka Consumer within the Kafka cluster. Kafka uses the Consumer group ID to track message offsets, so the consumer can resume collection from where it last left off. Only one Consumer group ID is allowed.	Required when using the New messages collection method.
Partition number	Note! This setting appears only when you select From offset. The specific Kafka partition number to start collection from. Enter the single partition number obtained from your Kafka broker.	Required when using the From offset collection method.
Starting offset	Note! This setting appears only when you select From offset. The specific offset number within the selected partition number to start collection from. Enter the single offset value obtained from your Kafka broker. Collection will begin from this offset and continue until the latest message in the partition.	Required when using the From offset collection method.
Note! The Kafka Consumer can collect messages in Buffer format and supports GZIP or ZSTD compression formats. The diagram below is an example of how the Kafka Consumer collects messages using both methods. Kafka Consumer offset diagram
Batch size
Messages to commit per batch	Set the number of messages you want to commit for each batch. If the topic contains 150 messages and the batch size is set to 100, the stream collects the first 100 messages from the topic to pass on. Then the next 50 messages are collected in the next batch, and the operation continues. If there are fewer messages than the batch size, all the messages are collected, and the stream continues with the next operation. The default value of the batch size is 100, which is also the minimum value. If you do not enter a value, then the default value will be used.	Required

Kafka Consumer

Kafka brokers

Authentication

Kafka topic

Collection method

Batch size