Skip to content

Commit c7d42d9

Browse files
committed
[WIP] SPARK-1903 Add initial port listing for documentation
1 parent a416ae9 commit c7d42d9

File tree

1 file changed

+80
-0
lines changed

1 file changed

+80
-0
lines changed

docs/configuration.md

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -791,3 +791,83 @@ you might compute `SPARK_LOCAL_IP` by looking up the IP of a specific network in
791791
Spark uses [log4j](http://logging.apache.org/log4j/) for logging. You can configure it by adding a
792792
`log4j.properties` file in the `conf` directory. One way to start is to copy the existing
793793
`log4j.properties.template` located there.
794+
795+
# Configuring ports for network security
796+
797+
Spark makes heavy use of the network, and some environments have strict requirements for using tight
798+
firewall settings. Below are the primary ports that Spark uses for its communication.
799+
800+
<table class="table">
801+
<tr>
802+
<th>From</th><th>To</th><th>Port</th><th>Purpose</th><th>Configuration
803+
Setting</th><th>Notes</th>
804+
</tr>
805+
<!-- Web UIs -->
806+
<tr>
807+
<td>Browser</td>
808+
<td>Master</td>
809+
<td>8080</td>
810+
<td>Web UI</td>
811+
<td><code>spark.history.ui.port</code></td>
812+
<td></td>
813+
</tr>
814+
<tr>
815+
<td>Browser</td>
816+
<td>Worker</td>
817+
<td>8081</td>
818+
<td>Web UI</td>
819+
<td><code>spark.ui.port</code></td>
820+
<td>Note: this is the same configuration setting as the webui on the driver</td>
821+
</tr>
822+
<tr>
823+
<td>Browser</td>
824+
<td>Driver</td>
825+
<td>4040</td>
826+
<td>Web UI</td>
827+
<td><code>spark.ui.port</code></td>
828+
<td>Note: this is the same configuration setting as the webui on the worker</td>
829+
</tr>
830+
831+
<!-- Cluster interactions -->
832+
<tr>
833+
<td>Application</td>
834+
<td>Master</td>
835+
<td>7077</td>
836+
<td>Submit job to cluster</td>
837+
<td><code>spark.driver.port</code></td>
838+
<td>Uses Akka. Set to "0" to choose a port randomly</td>
839+
</tr>
840+
<tr>
841+
<td>Worker</td>
842+
<td>Master</td>
843+
<td>7077</td>
844+
<td>Join cluster</td>
845+
<td><code>spark.driver.port</code></td>
846+
<td>Uses Akka. Set to "0" to choose a port randomly</td>
847+
</tr>
848+
<tr>
849+
<td>Application</td>
850+
<td>Worker</td>
851+
<td>(random)</td>
852+
<td>Join cluster</td>
853+
<td><code>SPARK_WORKER_PORT</code> (standalone cluster)</td>
854+
<td>Uses Akka</td>
855+
</tr>
856+
857+
<!-- Other misc stuff -->
858+
<tr>
859+
<td>Driver and other Workers</td>
860+
<td>Worker</td>
861+
<td>(random)</td>
862+
<td>
863+
<ul>
864+
<li>File server for file and jars</li>
865+
<li>Http Broadcast</li>
866+
<li>Class file server (Spark Shell only)</li>
867+
</ul>
868+
</td>
869+
<td>None</td>
870+
<td>Uses Jetty. Each of these services starts on a random port that cannot be configured</td>
871+
</tr>
872+
873+
</table>

0 commit comments

Comments
 (0)