Spark applications must be configured with a set of Kafka topics that are either shared between multiple applications or dedicated to specific applications. The assigned topics must be created before you submit an application to the Spark cluster. Before you can create the topics you must start Zookeeper and Kafka.
Prerequisites:
Prepare scripts according to 4.2 Preparing and Creating Scripts for KPI Management
Starting Clusters
To start a cluster follow the steps:
...
| Code Block | ||
|---|---|---|
| ||
$ ./bin/kafka-topics.sh --create --topic <input topic> --bootstrap-server \ localhost:9092 --partitions <number of partitions> --replication-factor <number of replicas> $ ./bin/kafka-topics.sh --create --topic <output topic> --bootstrap-server \ localhost:9092 --partitions <number of partitions> --replication-factor <number of replicas> $ ./bin/kafka-topics.sh --create --topic <alarm topic> --bootstrap-server \ localhost:9092 --partitions <number of partitions> --replication-factor <number of replicas> |
Example - Creating Kafka Topics
| Code Block |
|---|
./bin/kafka-topics.sh --create --topic kpi-output --partitions 6 --replication-factor 1 ./bin/kafka-topics.sh --create --topic kpi-input --partitions 6 --replication-factor 1 ./bin/kafka-topics.sh --create --topic kpi-alarm --partitions 6 --replication-factor 1 |
Example - Creating Kafka topics, overriding retention settings
| Code Block |
|---|
./bin/kafka-topics.sh --create --topic kpi-output --partitions 6 --replication-factor 1 --config retention.ms=86400000 ./bin/kafka-topics.sh --create --topic kpi-input --partitions 6 --replication-factor 1 --config retention.ms=86400000 ./bin/kafka-topics.sh --create --topic kpi-alarm --partitions 6 --replication-factor 1 --config retention.ms=86400000 |
...