Kafka Profile
In the Kafka profile configuration, you define which Kafka broker to consume messages from or produce messages to, if security should be applied, and if other specific settings for communication are needed.
Configuration
To create a new Kafka profile configuration, click the New Configuration button and select Kafka Profile from the Configurations dialog.
The Kafka profile configuration contains two tabs; Connectivity and Advanced.
Connectivity tab
The Connectivity tab is displayed by default when creating or opening a Kafka profile.
The Connectivity tab contains the following settings:
Setting | Description |
|---|---|
Host | The hostname of the Kafka broker. |
Port | The port of the Kafka broker. |
Security Profile | If you want to have a secure connection, click Browse… to select a Security profile with the certificate and configuration you want to use, see Security Profile. |
Advanced tab
In the Advanced tab, you can configure properties for optimizing the performance of the Kafka Producer and Consumer. The Advanced tab contains two tabs; Producer and Consumer.
Producer tab
In the Producer tab, you can configure the properties of the Kafka forwarding agent.
For information on how to configure the properties for SSL and Kerberos, see https://docs.cloudera.com/?tab=cdp-public-cloud
Note!
Once you have edited the JAAS file required for Kerberos, restart the EC to register the changes made.
Note!
If you make any changes to the security configuration of the Kafka Producer, any topics in use must be recreated before they can be used.
For further information on the properties, see https://kafka.apache.org.
Enabling compression for Kafka
Compression for messages sent to Kafka brokers can now be enabled from the Advanced producer properties. The compression codec utilized by the system follows the standard Kafka library, where Gzip, Lz4, and Snappy are supported.
To enable compression, just add the property compression.type into the Advanced producer properties and add the value, gzip, lz4, snappy or, none.
Retries and Duplicate Prevention
In the Kafka Forwarding Agents, retries do not result in duplicate records being written to the topic. This behavior is provided by the Kafka Forwarding Agent’s producer, which uses Kafka’s idempotent producer mode (enable.idempotence=true) and cannot be changed.
With idempotence enabled, the Kafka producer assigns a unique sequence number to each record it sends. If a record must be re‑sent because of a transient failure, for example, a network timeout or temporary broker unavailability, the broker compares the sequence number of the incoming record with previously received records. If the broker has already accepted a record with the same sequence number, it discards the duplicate.
Because idempotence is always enabled, setting the retries property to a value greater than 0 does not produce duplicate records as a result of retry attempts. The Kafka broker’s built‑in duplicate detection ensures that re‑sent records that have already been accepted are discarded rather than written again.
The retries property
The retries property controls how many times the Kafka producer attempts to resend a record that failed to be delivered. You configure this property in the Producer tab of the Advanced section in the Kafka Profile. The agent does not override the retries value configured in the profile: it uses either the value you configure or the default if you do not configure it.
Setting | Behavior |
|---|---|
Default value | If you do not set the retries property in the Kafka profile, the agent uses a default value of 15. |
retries > 0 | When |
retries = 0 | When |
Consumer tab
In the Consumer tab, you can configure the properties of the Kafka collection agent. See the text in the Advanced consumer properties field for further information about the properties.