Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

The Kafka Profile configuration defines which Kafka broker to consume from or produce to, if any security shall be applied and if other specific settings for communication is needed.

Configuration

To create a new Kafka profile configuration, click the New Configuration button and then select Kafka Profile from the Configurations dialog.

The Kafka profile configuration contains two tabs: Connectivity and Advanced.

Connectivity tab

The Connectivity tab is displayed by default when creating or opening a Kafka profile.

kafkaProfile.png

The Connectivity tab contains the following settings:

Setting

Description

Host

Hostname of the Kafka broker.

Port

Port of the Kafka broker.

Security Profile

Click Browse to select a security profile with certificate and configuration to use, if you prefer to use a secure connection. Refer to Security Profile for more information.

Advanced tab

In the Advanced tab, you can configure properties for optimizing the performance of the Kafka Producer and Consumer. The Advanced tab contains two tabs: Producer and Consumer.

Producer tab

In the Producer tab, you can configure the properties of the Kafka forwarding agent. 

kafkaProfile_advanced.png

Kafka profile configuration - Producer tab in the Advanced tab

The property producer.abortunknown=true sets the agent to abort if the broker replies with Unknown topic or partition.

When running in Acknowledged execution mode, the property producer.full.response determines if the data sent to the Kafka log is also included in the response UDR. The value is set to true by default. Setting the value to false reduces the memory footprint.

For information on how to configure the properties for SSL and Kerberos, please refer to https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html .

Note!

Once you have edited the JAAS file required for Kerberos, you will need to restart the EC to register the changes made.

Note!

If you make any changes to the security configuration of the Kafka Producer, any topics used must be recreated before they can be used.

For further information on the properties refer to https://kafka.apache.org .

Enabling Compression for Kafka

Compression for messages to be sent to Kafka brokers can now be enabled from the Advanced producer properties. The compression codec utilized by the system follows the standard Kafka library, where Gzip, Lz4 and Snappy are supported.

To enable compression, just add the property compression.type into the Advanced producer properties. Followed by the value, gzip, lz4, snappy or none.

Consumer tab

In the Consumer tab, you can configure the properties of the Kafka collection agent.

kafkaProfile_advancedConsumer.png

See the text in the Advanced consumer properties field for further information about the properties

  • No labels