Submitting a Spark Application
- Kevin Wu
This section describes how to submit a Spark application.
Prerequisite
The scripts must be prepared according to 5.2 Preparing and Creating Scripts for KPI Management.
Submitting the Spark Cluster
Use this script to submit a Spark application to the Spark workers:
$ submit.sh <kpiapp name>Where the kpiapp name must match a name present in the script kpi_params.sh in the same folder. Default is:
submit.sh kpiapp
You can now confirm the status of the application. Open a browser and go to:
http://<spark service master host>:8080Â
This opens the Spark Master Web UI.
Spark UI
Hint!
The Application Detail UI contains useful tools for monitoring of the submitted applications. This UI can be opened from the Spark Master Web UI by clicking the Application Name of a running application.
Click the top-level menus in the UI to monitor various aspects of the submitted applications. The Scheduling Delay and Processing Time graphs under the Streaming menu are particularly useful for monitoring the processing performance. The former displays the scheduling time of "Spark batches" and should be near 0. The latter display the processing time of these batches.