@@ -791,3 +791,83 @@ you might compute `SPARK_LOCAL_IP` by looking up the IP of a specific network in
791791Spark uses [ log4j] ( http://logging.apache.org/log4j/ ) for logging. You can configure it by adding a
792792` log4j.properties ` file in the ` conf ` directory. One way to start is to copy the existing
793793` log4j.properties.template ` located there.
794+
795+ # Configuring ports for network security
796+
797+ Spark makes heavy use of the network, and some environments have strict requirements for using tight
798+ firewall settings. Below are the primary ports that Spark uses for its communication.
799+
800+ <table class =" table " >
801+ <tr >
802+ <th>From</th><th>To</th><th>Port</th><th>Purpose</th><th>Configuration
803+ Setting</th><th>Notes</th>
804+ </tr >
805+ <!-- Web UIs -->
806+ <tr >
807+ <td>Browser</td>
808+ <td>Master</td>
809+ <td>8080</td>
810+ <td>Web UI</td>
811+ <td><code>spark.history.ui.port</code></td>
812+ <td></td>
813+ </tr >
814+ <tr >
815+ <td>Browser</td>
816+ <td>Worker</td>
817+ <td>8081</td>
818+ <td>Web UI</td>
819+ <td><code>spark.ui.port</code></td>
820+ <td>Note: this is the same configuration setting as the webui on the driver</td>
821+ </tr >
822+ <tr >
823+ <td>Browser</td>
824+ <td>Driver</td>
825+ <td>4040</td>
826+ <td>Web UI</td>
827+ <td><code>spark.ui.port</code></td>
828+ <td>Note: this is the same configuration setting as the webui on the worker</td>
829+ </tr >
830+
831+ <!-- Cluster interactions -->
832+ <tr >
833+ <td>Application</td>
834+ <td>Master</td>
835+ <td>7077</td>
836+ <td>Submit job to cluster</td>
837+ <td><code>spark.driver.port</code></td>
838+ <td>Uses Akka. Set to "0" to choose a port randomly</td>
839+ </tr >
840+ <tr >
841+ <td>Worker</td>
842+ <td>Master</td>
843+ <td>7077</td>
844+ <td>Join cluster</td>
845+ <td><code>spark.driver.port</code></td>
846+ <td>Uses Akka. Set to "0" to choose a port randomly</td>
847+ </tr >
848+ <tr >
849+ <td>Application</td>
850+ <td>Worker</td>
851+ <td>(random)</td>
852+ <td>Join cluster</td>
853+ <td><code>SPARK_WORKER_PORT</code> (standalone cluster)</td>
854+ <td>Uses Akka</td>
855+ </tr >
856+
857+ <!-- Other misc stuff -->
858+ <tr >
859+ <td>Driver and other Workers</td>
860+ <td>Worker</td>
861+ <td>(random)</td>
862+ <td>
863+ <ul>
864+ <li>File server for file and jars</li>
865+ <li>Http Broadcast</li>
866+ <li>Class file server (Spark Shell only)</li>
867+ </ul>
868+ </td>
869+ <td>None</td>
870+ <td>Uses Jetty. Each of these services starts on a random port that cannot be configured</td>
871+ </tr >
872+
873+ </table >
0 commit comments