Skip to content

[SPARK-22648][K8s] Add documentation covering init containers and secrets #20059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liyinan926 wants to merge 7 commits into from
Closed

[SPARK-22648][K8s] Add documentation covering init containers and secrets #20059

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0a659a
[SPARK-22648][Kubernetes] Update documentation to cover features in #…
liyinan926 Dec 22, 2017
fbb2112
Addressed comments
liyinan926 Dec 22, 2017
f23bf0f
Addressed more comments
liyinan926 Dec 23, 2017
818abaf
Update the unit of one configuration property
liyinan926 Dec 25, 2017
08486e8
Fixed the default value of a config property
liyinan926 Dec 26, 2017
f4b5c03
Addressed more comments
liyinan926 Dec 27, 2017
453a3db
Fixed some formatting
liyinan926 Dec 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 109 additions & 49 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application dependencies can be pre-moun
Those dependencies can be added to the classpath by referencing them with `local://` URIs and/or setting the
`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.

### Using Remote Dependencies
When there are application dependencies hosted in remote locations like HDFS or HTTP servers, the driver and executor pods need a Kubernetes [init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) for downloading the dependencies so the driver and executor containers can use them locally. This requires users to specify the container image for the init-container using the configuration property `spark.kubernetes.initContainer.image`. For example, users simply add the following option to the `spark-submit` command to specify the init-container image:
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and below text should be broken up into multiple lines.

Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should include 2-3 examples of remote file usage - ideally, showing that one can use http, hdfs, gcs, s3 in dependencies.

Copy link
Contributor Author

@liyinan926 liyinan926 Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to break them into lines? I thought this should be automatically wrapped when being viewed.

Copy link
Contributor Author

@liyinan926 liyinan926 Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding examples, I can add one spark-submit example showing how to use remote jars/files on http/https and hdfs. But gcs requires the connector in the init-container, which is non-trivial. I'm not sure about s3. I think we should avoid doing so.

Copy link
Contributor

@foxish foxish Dec 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HDFS or HTTP sound good. We can cover GCS elsewhere. Line breaks were for ease of reviewing by others (being able to comment on individual lines) and for consistency with the rest of the docs.


```
--conf spark.kubernetes.initContainer.image=<init-container image>
```

## Secret Management
In some cases, a Spark application may need to use some credentials, e.g., for accessing data on a secured HDFS cluster or cloud storage that requires users to provide credentials for authentication. This can be done by mounting the credentials into the driver and executor containers using Kubernetes [secrets](https://kubernetes.io/docs/concepts/configuration/secret/). To mount a user-specified secret into the driver container, users can use the configuration property of the form `spark.kubernetes.driver.secrets.[SecretName]=<mount path>`. Similarly, the configuration property of the form `spark.kubernetes.executor.secrets.[SecretName]=<mount path>` can be used to mount a user-specified secret into the executor containers. Note that it is assumed that the secret to be mounted is in the same namespace as that of the driver and executor pods. For example, to mount a secret named `spark-secret` onto the path `/etc/secrets` in both the driver and executor containers, add the following options to the `spark-submit` command:

```
--conf spark.kubernetes.driver.secrets.spark-secret=/etc/secrets
--conf spark.kubernetes.executor.secrets.spark-secret=/etc/secrets
```

Note that if an init-container is used, any secret mounted into the driver container will also be mounted into the init-container of the driver. Similarly, any secret mounted into an executor container will also be mounted into the init-container of the executor.

## Introspection and Debugging

These are the different ways in which you can investigate a running/completed Spark application, monitor progress, and
Expand Down Expand Up @@ -275,7 +292,7 @@ specific to Spark on Kubernetes.
<td><code>(none)</code></td>
<td>
Container image to use for the driver.
This is usually of the form `example.com/repo/spark-driver:v1.0.0`.
This is usually of the form <code>example.com/repo/spark-driver:v1.0.0</code>.
This configuration is required and must be provided by the user.
</td>
</tr>
Expand All @@ -284,7 +301,7 @@ specific to Spark on Kubernetes.
<td><code>(none)</code></td>
<td>
Container image to use for the executors.
This is usually of the form `example.com/repo/spark-executor:v1.0.0`.
This is usually of the form <code>example.com/repo/spark-executor:v1.0.0</code>.
This configuration is required and must be provided by the user.
</td>
</tr>
Expand Down Expand Up @@ -528,51 +545,94 @@ specific to Spark on Kubernetes.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driver.limit.cores</code></td>
<td>(none)</td>
<td>
Specify the hard CPU [limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) for the driver pod.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.executor.limit.cores</code></td>
<td>(none)</td>
<td>
Specify the hard CPU [limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) for each executor pod launched for the Spark Application.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
<td>(none)</td>
<td>
Adds to the node selector of the driver pod and executor pods, with key <code>labelKey</code> and the value as the
configuration's value. For example, setting <code>spark.kubernetes.node.selector.identifier</code> to <code>myIdentifier</code>
will result in the driver pod and executors having a node selector with key <code>identifier</code> and value
<code>myIdentifier</code>. Multiple node selector keys can be added by setting multiple configurations with this prefix.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
<td>(none)</td>
<td>
Add the environment variable specified by <code>EnvironmentVariableName</code> to
the Driver process. The user can specify multiple of these to set multiple environment variables.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.mountDependencies.jarsDownloadDir</code></td>
<td><code>/var/spark-data/spark-jars</code></td>
<td>
Location to download jars to in the driver and executors.
This directory must be empty and will be mounted as an empty directory volume on the driver and executor pods.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.mountDependencies.filesDownloadDir</code></td>
<td><code>/var/spark-data/spark-files</code></td>
<td>
Location to download jars to in the driver and executors.
This directory must be empty and will be mounted as an empty directory volume on the driver and executor pods.
</td>
</tr>
<td><code>spark.kubernetes.driver.limit.cores</code></td>
<td>(none)</td>
<td>
Specify the hard CPU [limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) for the driver pod.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.executor.limit.cores</code></td>
<td>(none)</td>
<td>
Specify the hard CPU [limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) for each executor pod launched for the Spark Application.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
<td>(none)</td>
<td>
Adds to the node selector of the driver pod and executor pods, with key <code>labelKey</code> and the value as the
configuration's value. For example, setting <code>spark.kubernetes.node.selector.identifier</code> to <code>myIdentifier</code>
will result in the driver pod and executors having a node selector with key <code>identifier</code> and value
<code>myIdentifier</code>. Multiple node selector keys can be added by setting multiple configurations with this prefix.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
<td>(none)</td>
<td>
Add the environment variable specified by <code>EnvironmentVariableName</code> to
the Driver process. The user can specify multiple of these to set multiple environment variables.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.mountDependencies.jarsDownloadDir</code></td>
<td><code>/var/spark-data/spark-jars</code></td>
<td>
Location to download jars to in the driver and executors.
This directory must be empty and will be mounted as an empty directory volume on the driver and executor pods.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.mountDependencies.filesDownloadDir</code></td>
<td><code>/var/spark-data/spark-files</code></td>
<td>
Location to download jars to in the driver and executors.
This directory must be empty and will be mounted as an empty directory volume on the driver and executor pods.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.mountDependencies.mountTimeout</code></td>
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should rename this to mountDependencies.timeout to avoid reiterating "mount"

<td>5 minutes</td>
<td>
Timeout before aborting the attempt to download and unpack dependencies from remote locations when initializing
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be more precise on what operation is happening.
"initializing" -> "when downloading and unpacking dependencies into"

the driver and executor pods.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.initContainer.image</code></td>
<td>(none)</td>
<td>
Container image for the init-container of the driver and executors for downloading dependencies.
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link to init-container docs.

This is usually of the form <code>example.com/repo/spark-init:v1.0.0</code>.
This configuration is optional and must be provided by the user if any non-container local dependency is used and
must be downloaded remotely.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.initContainer.maxThreadPoolSize</code></td>
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this option name needs fixing. Maybe spark.kubernetes.mountDependencies.maxThreadPoolSize?

<td>5</td>
<td>
Maximum size of the thread pool in the init-container for downloading remote dependencies.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driver.secrets.[SecretName]</code></td>
<td>(none)</td>
<td>
Add the secret named <code>SecretName</code> to the driver pod on the path specified in the value. For example,
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

secret -> Kubernetes Secret
Please also link to the Kubernetes secrets docs page.

<code>spark.kubernetes.driver.secrets.spark-secret=/etc/secrets</code>. Note that if an init-container is used,
the secret will also be add to the init-container in the driver pod.
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/add/added/

</td>
</tr>
<tr>
<td><code>spark.kubernetes.executor.secrets.[SecretName]</code></td>
<td>5</td>
Copy link
Contributor

@jiangxb1987 jiangxb1987 Dec 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the meaning of 5 here?

Copy link
Contributor Author

@liyinan926 liyinan926 Dec 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, copy and paste error. Fixed.

<td>
Add the secret named <code>SecretName</code> to the executor pod on the path specified in the value. For example,
<code>spark.kubernetes.executor.secrets.spark-secret=/etc/secrets</code>. Note that if an init-container is used,
the secret will also be add to the init-container in the executor pod.
Copy link
Contributor

@foxish foxish Dec 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/add/added/

</td>
</tr>
</table>
3 changes: 2 additions & 1 deletion sbin/build-push-docker-images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
# with Kubernetes support.

declare -A path=( [spark-driver]=kubernetes/dockerfiles/driver/Dockerfile \
[spark-executor]=kubernetes/dockerfiles/executor/Dockerfile )
[spark-executor]=kubernetes/dockerfiles/executor/Dockerfile \
[spark-init]=kubernetes/dockerfiles/init-container/Dockerfile )

function build {
docker build -t spark-base -f kubernetes/dockerfiles/spark-base/Dockerfile .
Expand Down