Skip to content

[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness #25765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 8 commits into from
+72 −19
Closed

[SPARK-28937][SPARK-28936][KUBERNETES] Reduce test flakyness #25765

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e9bf47f
Work on debugging the secrets test
holdenk Aug 28, 2019
805fe3a
Test pass (yay)
holdenk Aug 30, 2019
86f9e75
Add a listener in place of Thread.sleep for executing commands inside…
holdenk Aug 30, 2019
3ccddbf
Remove printlns from my debugging
holdenk Aug 31, 2019
916372c
Switch some constant uses of username/password to the object ref and …
holdenk Sep 4, 2019
16ba894
cammel fix
holdenk Sep 11, 2019
2ab68b5
Style fixes
holdenk Sep 11, 2019
de35d67
Add missing Locale import
holdenk Sep 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Switch some constant uses of username/password to the object ref and … 
…do a more complete env check
  • Loading branch information
@holdenk
holdenk committed Sep 4, 2019
commit 916372caa2241deafffd81e40f3dd89d08e162b9
18 changes: 12 additions & 6 deletions ...-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/SecretsTestsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,17 @@ private[spark] trait SecretsTestsSuite { k8sSuite: KubernetesSuite =>
createTestSecret()
sparkAppConf
.set(s"spark.kubernetes.driver.secrets.$ENV_SECRET_NAME", SECRET_MOUNT_PATH)
.set(s"spark.kubernetes.driver.secretKeyRef.USERNAME", s"$ENV_SECRET_NAME:username")
.set(s"spark.kubernetes.driver.secretKeyRef.PASSWORD", s"$ENV_SECRET_NAME:password")
.set(
s"spark.kubernetes.driver.secretKeyRef.${ENV_SECRET_KEY_1.toUpperCase}",
s"$ENV_SECRET_NAME:${ENV_SECRET_KEY_1}")
.set(
s"spark.kubernetes.driver.secretKeyRef.${ENV_SECRET_KEY_2.toUpperCase}",
s"$ENV_SECRET_NAME:${ENV_SECRET_KEY_2}")
.set(s"spark.kubernetes.executor.secrets.$ENV_SECRET_NAME", SECRET_MOUNT_PATH)
.set(s"spark.kubernetes.executor.secretKeyRef.USERNAME", s"$ENV_SECRET_NAME:username")
.set(s"spark.kubernetes.executor.secretKeyRef.PASSWORD", s"$ENV_SECRET_NAME:password")
.set(s"spark.kubernetes.executor.secretKeyRef.USERNAME",
s"${ENV_SECRET_NAME}:$ENV_SECRET_KEY_1")
.set(s"spark.kubernetes.executor.secretKeyRef.PASSWORD",
s"${ENV_SECRET_NAME}:$ENV_SECRET_KEY_2")
try {
runSparkPiAndVerifyCompletion(
driverPodChecker = (driverPod: Pod) => {
Expand Down Expand Up @@ -91,8 +97,8 @@ private[spark] trait SecretsTestsSuite { k8sSuite: KubernetesSuite =>
assert(!env.isEmpty)
env
}
env.toString should include (ENV_SECRET_VALUE_1)
env.toString should include (ENV_SECRET_VALUE_2)
env.toString should include (s"${ENV_SECRET_KEY_1.toUpperCase}=$ENV_SECRET_VALUE_1")
env.toString should include (s"${ENV_SECRET_KEY_2.toUpperCase}=$ENV_SECRET_VALUE_2")

// Make sure our secret files are mounted correctly
val files = Utils.executeCommand("ls", s"$SECRET_MOUNT_PATH")
Expand Down