Kafka Producer

The purpose of this forwarder is to send records (as messages) to Kafka brokers with ease. With Kafka Producer, you send records to one or more topics in configured Kafka brokers. The partition method and compression of the records can also be configured. The records are sent to the Kafka broker in the JSON format.

You can configure this function in the following settings. All these configurations are mandatory.

Kafka brokers to set the Kafka broker to send the record to.
Authentication is to set the secret from Secrets Wallet with the credentials of the Kafka broker.
Kafka Topic to select the method the record is sent to the Kafka broker.
Partition setting to select the partition method to send the records to.
Gzip compression to compress the records.

Kafka brokers

Fill in the server host IP and the port number of the Kafka broker that you will want your stream to send the records to.

Field	Description

Field	Description
Host	The hostname, IPv4 or IPv6 address of the Kafka broker.
Port	The listening port of the Kafka broker. The default port is 9092

You can add additional Kafka brokers by clicking on Add broker.

Authentication

Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Wallet on how to configure a secret using the Secrets Wallet.

If the Kafka broker does not have any authentication configured, you may disable Authentication by toggling it off.

Note!

SSL is enabled for all authentication types.

Kafka Topic

Select the method the records are sent to the Kafka broker by checking one of the following options.

Settings	Description

Settings

Description

Add topic name

Enter the topic name in Topic name to send the records to. You may add more than one topic in Topic name.

Note!

Ensure to enter the full and correct topic name. Topic names are also case sensitive.

If the topic name does not exist in the broker, the topic is automatically created with partitions following the broker’s settings.

Select topic field

Select a field that contains the topics from the Topic field drop-down to send the records to the selected topic.

If the field does not exist in the broker, you can add the field by clicking Add field.

Partition setting

Provides options to select the partitions where the records are sent to.

Option	Description

Option	Description
Round-robin	Distributes all records in a round-robin fashion to all available partitions. This is the default selection.
Partition key	Sends the records to specific partitions based on the partition key. A partition is assigned based on a hash generated from the partition key. The same key will always be assigned the same partition. Select the partition key field from the drop-down in Partition key field, or add a new field by clicking Add field.
Partition ID	Sends the records to specific partitions based on the partition ID. The assigned partition will be taken directly from the partition ID provided. Select the partition ID field from the drop-down in Partition ID field, or add a new field by clicking Add field. Note! Non-existent partition IDs results in stream failure.

Gzip compression

You can toggle on Gzip compression to compress the records before sending to the Kafka broker. Compressed records will take up less storage in the broker.