You can set the Spark ports directly in the Spark. The available Spark ports are defined in the table below.
When you are running the Spark cluster, the following network ports will be used:
Port Name | Default Port Number | Description |
|---|---|---|
Spark Master Port | 7077 | This port is used by the Spark master for communication with Spark workers, and |
KPI Management. To change port number, set the property | ||
Spark Worker Port | Dynamic Port Range/Random | This port is used by the Spark worker. To change port number,set the property SPARK_WORKER_PORT in the spark-environment block in the Spark service configuration |
Spark Master Web UI Port | 8080 | This port is used by the Spark master Web UI. To change port number, set the property MASTER_WEBUI_PORT in the spark-environment block in the Spark service configuration |
Spark Worker Web UI Port | 8081 | This port is used by the Spark worker Web UI. To change port number,set the property WORKER_WEBUI_PORT in the spark-environment block in the Spark service configuration |
Spark UI Port | 4040 | Every SparkContext launches a web UI that displays useful information about the application. If multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc)". To change that initial default port set the SPARK_UI_PORT in the spark-environment block of the spark service context. To change the ui port for specific application set the property spark.ui.port in the spark app block |
Spark Driver Port | Dynamic Port Range/Random | By default, the Spark drivers use random ports for each submitted Spark application, to communicate with the Spark masters and the executors. To bind the drivers to a specific port number, set the driver port in the spark-conf block of the specific spark app block. Example of a port setting: |
Block Manager Port | Dynamic Port Range/Random | By default, the block managers that runs the drivers on each executor, use random ports to communicate with each other. To bind the block managers to a specific port number, set the block manager port |
in the spark-conf block of the specific spark app block. Example of a port setting: | |||
Spark REST Port | 6066 | This port is used by the Spark REST server interface, which is currently not included in the Apache Spark documentation.
|
|