Redeploying Spark Workers

Use the provided scripts stop.sh and start.sh to redeploy the Spark workers. This is required after you have updated the number of worker nodes in the spark cluster. increased or decreased the number of slave Service Contexts that are available to the spark service.

$ stop.sh

$ start_master_workers.sh

When you change the number of spark slaves, you may also want to update the property spark.default.parallellism in the service configuration. This is to optimize the performance of the Spark cluster. However, for the change in the Spark configuration to become effective, you must restart the cluster and submit the spark application. For further information about spark.default.parallellism, see KPI Management - External Software.

If the Spark UI indicates that a worker has the status "DEAD", you must restart the cluster.