In the Kafka profile configuration, you define which Kafka broker to consume messages from or produce messages to, if security should be applied, and if other specific settings for communication are needed.

Configuration

To create a new Kafka profile configuration, click the New Configuration button and select Kafka Profile from the Configurations dialog.

The Kafka profile configuration contains two tabs; Connectivity and Advanced.

Connectivity tab

The Connectivity tab is displayed by default when creating or opening a Kafka profile.

Kafka profile configuration - Connectivity tab

The Connectivity tab contains the following settings:

Setting	Description

Setting	Description
Host	The hostname of the Kafka broker.
Port	The port of the Kafka broker.
Security Profile	If you want to have a secure connection, click Browse… to select a Security profile with the certificate and configuration you want to use, see Security Profile.

Advanced tab

In the Advanced tab, you can configure properties for optimizing the performance of the Kafka Producer and Consumer. The Advanced tab contains two tabs; Producer and Consumer.

Producer tab

In the Producer tab, you can configure the properties of the Kafka forwarding agent.

Kafka profile configuration - Producer tab in the Advanced tab

For information on how to configure the properties for SSL and Kerberos, see Configuring Apache Kafka Security | 4.1.x | Cloudera Documentation.

Note!

Once you have edited the JAAS file required for Kerberos, restart the EC to register the changes made.

Note!

If you make any changes to the security configuration of the Kafka Producer, any topics in use must be recreated before they can be used.

For further information on the properties, see https://kafka.apache.org.

Enabling compression for Kafka

Compression for messages sent to Kafka brokers can now be enabled from the Advanced producer properties. The compression codec utilized by the system follows the standard Kafka library, where Gzip, Lz4, and Snappy are supported.

To enable compression, just add the property compression.type into the Advanced producer properties and add the value, gzip, lz4, snappy or, none.

Consumer tab

In the Consumer tab, you can configure the properties of the Kafka collection agent.

See the text in the Advanced consumer properties field for further information about the properties.