Kafka producer
- Casey Han
- Jenni Wallin
The purpose of this forwarder is to send records (as messages) to Kafka brokers with ease. With Kafka producer, you send records to one or more topics in configured Kafka brokers. The partition method and compression of the records can also be configured. The records are sent to the Kafka broker in the JSON format.
You can configure this function in the following settings. All these configurations are mandatory.
Kafka brokers to set the Kafka broker to send the record to.
Authentication is to set the secret from Secrets Wallet with the credentials of the Kafka broker.
Kafka topic to select the method the record is sent to the Kafka broker.
Partition setting to select the partition method to send the records to.
Gzip compression to compress the records.
Kafka brokers
Fill in the server host IP and the port number of the Kafka broker that you will want your stream to send the records to.
Field | Description |
|---|---|
Host | The hostname, IPv4 or IPv6 address of the Kafka broker. |
Port | The listening port of the Kafka broker. The default port is 9092 |
You can add additional Kafka brokers by clicking on Add broker.
Authentication
Select from a list of secrets in your Secrets Wallet that contain the authentication credentials for the Kafka brokers. For more information, see Secrets Wallet on how to configure a secret using the Secrets Wallet.
If the Kafka broker does not have any authentication configured, you may disable Authentication by toggling it off.
Note!
SSL is enabled for all authentication types.
Kafka topic
Select the method the records are sent to the Kafka broker by checking one of the following options.
Settings | Description |
|---|---|
Add topic name | Enter the topic name in Topic name to send the records to. You may add more than one topic in Topic name. Note! Ensure to enter the full and correct topic name. Topic names are also case sensitive. If the topic name does not exist in the broker, the topic is automatically created with partitions following the broker’s settings. |
Select topic field | Select a field that contains the topics from the Topic field drop-down to send the records to the selected topic. If the field does not exist in the broker, you can add the field by clicking Add field. |
Partition setting
Provides options to select the partitions where the records are sent to.
Option | Description |
|---|---|
Round-robin | Distributes all records in a round-robin fashion to all available partitions. This is the default selection. |
Partition key | Sends the records to specific partitions based on the partition key. A partition is assigned based on a hash generated from the partition key. The same key will always be assigned the same partition. Select the partition key field from the drop-down in Partition key field, or add a new field by clicking Add field. |
Partition ID | Sends the records to specific partitions based on the partition ID. The assigned partition will be taken directly from the partition ID provided. Select the partition ID field from the drop-down in Partition ID field, or add a new field by clicking Add field. Note! Non-existent partition IDs results in stream failure. |
Gzip compression
You can toggle on Gzip compression to compress the records before sending to the Kafka broker. Compressed records will take up less storage in the broker.