Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

This section describes the Kafka agents. The agents are used to forward and collect messages using Kafka. Kafka is a cluster-based software that is executed outside of. Kafka uses a high throughput publish-subscriber messaging model. It persists all messages and by design it decouples the publisher from the subscriber. That means that the forwarding agent will keep writing to the log even if the collection agents terminate.

The Kafka forwarding agent is listed among the processing agents in Desktop while the Kafka collection agent is listed among the collection agents.

Prerequisites

The reader of this document should be familiar with:

  • Apache Kafka

    Note!

    A working knowledge of Apache Kafka is required. For further information, see http://kafka.apache.org.

The Kafka agents enable you to configure workflows in with improved scalability and fault tolerance. As part of the data collection, data is written to Kafka to secure it from being lost if a failure occurs, and each topic can be set up to be replicated across several servers.

Example of Kafka workflows

Scaling

Using Kafka provides the capability to scale as required. One of the many ways to scale a Kafka cluster is when you create your Kafka configuration. It is recommended that when creating your Kafka configuration, you consider how many partitions you may eventually require and add more than you may  currently require,  as this will make it easier to scale up at a later stage.

Example of Scaling a Kafka Cluster


You can also refer to http://kafka.apache.org for guidance on scaling using partitions.

ItemDescription
<IP>:<port>

The IP and port of a Zookeeper node in the cluster. To find the connection details to all Zookeeper nodes, run the mzsh command service dump.

Example of service dump output

"zookeeper" : {
    "zookeeper1" : {
        "clientinfo" : {
            "clientinfo" : "10.46.124.46:5653,10.46.124.46:5703,10.46.124.46:5753"
        },

<service id>

The service id is the Kafka service name defined in custom.conf.


Example of custom.conf showing a service id


Example of reference to embedded Zookeeper service instance

bin/kafka-topics.sh --zookeeper 10.46.124.46:5653/kafka/kafka1 --list

is the equivalent of

mzsh mzadmin/dr kafka --service-key kafka1 --list

For information on how to use the different tools delivered with Kafka, see http://kafka.apache.org.

Authentication with Kerberos and Encryption with SSL

You can configure the Kafka profile to use authentication via Kerberos and SSL encryption. Supported properties for this are available in the Producer and Consumer tabs in the Kafka profile. See https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html for further information on how to configure this.


The section contains the following subsections:

  • No labels