Skip to content

[SPARK-24090][K8S] Update running-on-kubernetes.md #22224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liyinan926 wants to merge 2 commits into from
Closed

[SPARK-24090][K8S] Update running-on-kubernetes.md #22224

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48ecb1e
[SPARK-24090][K8S] Update running-on-kubernetes.md
liyinan926 Aug 24, 2018
7e8144b
ddressed review comments
liyinan926 Aug 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-24090][K8S] Update running-on-kubernetes.md
  • Loading branch information
@liyinan926
liyinan926 committed Aug 24, 2018
commit 48ecb1eac55d625361c0e1f52cde15079055ede5
39 changes: 31 additions & 8 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,35 @@ To use a secret through an environment variable use the following options to the
--conf spark.kubernetes.executor.secretKeyRef.ENV_NAME=name:key
```

## Using Kubernetes Volumes
Starting Spark 2.4.0, users can mount the following types of Kubernetes [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) into the driver and executor pods:
Copy link
Member

@srowen srowen Aug 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Starting with Spark ...". Also put a blank line above this.

Copy link
Contributor Author

@liyinan926 liyinan926 Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath): mounts a file or directory from the host node’s filesystem into a pod.
* [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir): an initially empty volume created when a pod is assigned to a node.
* [persistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim): used to mount a `PersistentVolume` into a pod.

To mount a volume of any of the types above into the driver pod, use the following configuration property:

```
--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path=<mount path>
--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly=<true|false>
```

Specifically, `VolumeType` can be one of the following values: `hostPath`, `emptyDir`, and `persistentVolumeClaim`. `VolumeName` is the name you want to use for the volume under the `volumes` field in the pod specification.

Each supported type of volumes may have some specific configuration options, which can be specified using configuration properties of the following form:

```
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName]=<value>
```

For example, the claim name of a `persistentVolumeClaim` with volume name `checkpointpvc` can be specified using the following property:

```
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=check-point-pvc-claim
```

The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`. For a complete list of available options for each supported type of volumes, please refer to the [Spark Properties](#spark-properties) section below.

## Introspection and Debugging

These are the different ways in which you can investigate a running/completed Spark application, monitor progress, and
Expand Down Expand Up @@ -299,21 +328,15 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl

## Future Work

There are several Spark on Kubernetes features that are currently being incubated in a fork -
[apache-spark-on-k8s/spark](https://github.com/apache-spark-on-k8s/spark), which are expected to eventually make it into
future versions of the spark-kubernetes integration.
There are several Spark on Kubernetes features that are currently being worked on or planned to be worked on. Those features are expected to eventually make it into future versions of the spark-kubernetes integration.
Copy link
Member

@srowen srowen Aug 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine; could just be removed too.

Copy link
Contributor Author

@liyinan926 liyinan926 Aug 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You meant removing the entire Future Work section or just this sentence?

Copy link
Member

@srowen srowen Aug 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just these two sentences, not the whole section. I figured it goes without saying there are features that may be worked on the the future that may go into Spark. But I don't feel strongly about it; this is fine.


Some of these include:

* R
* Dynamic Executor Scaling
* Dynamic Resource Allocation and External Shuffle Service
* Local File Dependency Management
* Spark Application Management
* Job Queues and Resource Management

You can refer to the [documentation](https://apache-spark-on-k8s.github.io/userdocs/) if you want to try these features
and provide feedback to the development team.

# Configuration

See the [configuration page](configuration.html) for information on Spark configurations. The following configurations are
Expand Down