To be able to handle the KPI management system in a Private Container Deployment, such as Kubernetes, you must prepare a number of scripts according to the instructions below. The scripts that you create are the following:
flush.sh
kpi_params.sh
spark_common_param.sh
start_master_workers.sh
stop.sh
submit.sh
These scripts will be used by different procedures that you find in the sections for KPI management - Distributed Processing.
Preparations before creating scripts:
A Prerequisite is that Spark, ZooKeeper, and Kafka are installed and up and running.
Creating scripts:
1. Set up your preferred KPI configuration or use the simplified example configuration, startup the platform. kpi_tst.zip
2. Find and copy the kpi_spark*.mzp among the installation files. Copy it to a place you want to keep your kpi application files.
3. To install the KPI app after building it, and extract the app installation:
Code Block |
---|
$ cd build/libs $ java -jar kpi_spark_8.1.1.0-SNAPSHOT.mzp install |
4. You will find the new directory mz_kpiapp that contains all app software.
Code Block |
---|
$ ls -l mz_kpiapp/, will list: app # The MZ kpi app bin # Shell script to handle the app jars # Extra jar files for the app |
5. Move the mz_kpiapp folder, to where you prefer to have it and add it to the PATH environment variable.
Code Block |
---|
Example: $ mv mz_kpiapp ~/ $ export PATH=$PATH:/home/user/mz_kpiapp/bin |
6. Set the environment variable SPARK_HOME.
Code Block |
---|
$ export SPARK_HOME="your spark home" |
7. Edit the kpiapp/bin/spark_common_param.sh, so it has the SPARK_HOME path.
Starting KPI
Note |
---|
Prerequisite Before you continue: Spark applications must be configured with a set of Kafka topics that are either shared between multiple applications or dedicated to specific applications. The assigned topics must be created before you submit an application to the Spark service. Before you can create the topics you must start the Kafka and Zookeeper services. |
8. Startup Spark cluster:
Code Block |
---|
$ start_master_workers.sh ... |
9. Submit the app:
Code Block |
---|
$ submit.sh kpiapp ... |
10. You now can see 2 workers, and 2 executors:
Code Block |
---|
$ jps Will give you something like: pid1 Worker pid2 Worker pid3 CoarseGrainedExecutorBackend pid4 CoarseGrainedExecutorBackend pid5 DriverWrapper pid6 CodeServerMain pid8 Master |
...