Skip to content

[SPARK-46058][CORE] Add separate flag for privateKeyPassword #43998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hasnain-db wants to merge 6 commits into from
Closed

[SPARK-46058][CORE] Add separate flag for privateKeyPassword #43998

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fd315a2
working
hasnain-db Nov 22, 2023
bcfea46
fix lint
hasnain-db Nov 22, 2023
de3cb43
fix lint, again
hasnain-db Nov 22, 2023
36dd5a2
fix tests
hasnain-db Nov 22, 2023
5cd6985
doc typo
hasnain-db Nov 28, 2023
312f518
fix comment
hasnain-db Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@ private SSLFactory createSslFactory() {
.requestedCiphers(conf.sslRpcRequestedCiphers())
.keyStore(conf.sslRpcKeyStore(), conf.sslRpcKeyStorePassword())
.privateKey(conf.sslRpcPrivateKey())
.privateKeyPassword(conf.sslRpcPrivateKeyPassword())
.keyPassword(conf.sslRpcKeyPassword())
.certChain(conf.sslRpcCertChain())
.trustStore(
Expand Down
18 changes: 15 additions & 3 deletions common/network-common/src/main/java/org/apache/spark/network/ssl/SSLFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ private void initNettySslContexts(final Builder b)
.build();

nettyServerSslContext = SslContextBuilder
.forServer(b.certChain, b.privateKey, b.keyPassword)
.forServer(b.certChain, b.privateKey, b.privateKeyPassword)
Copy link
Contributor

@JoshRosen JoshRosen Nov 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense and I agree that this seems like the right thing to do for PEM-based authentication.

After this change, though, I don't see any additional usages of b.keyPassword in RPL SSL code.

That code was added in cfea300.

There, it looks like it influences both Akka and Jetty.

In the Jetty code, which hasn't changed to this day, it does

      keyStorePassword.foreach(sslContextFactory.setKeyStorePassword)
      keyPassword.foreach(sslContextFactory.setKeyManagerPassword)

and flows to the "key manager" password.

Within Jetty, that password is read at once place:

protected KeyManager[] getKeyManagers(KeyStore keyStore) throws Exception
    {
        KeyManager[] managers = null;

        if (keyStore != null)
        {
            KeyManagerFactory keyManagerFactory = getKeyManagerFactoryInstance();
            keyManagerFactory.init(keyStore, _keyManagerPassword == null ? (_keyStorePassword == null ? null : _keyStorePassword.toString().toCharArray()) : _keyManagerPassword.toString().toCharArray());
            managers = keyManagerFactory.getKeyManagers();

So based this Jetty code, it looks like the right place to thread this password is to use it when initializing the KeyManagerFactory.

Given that, do you think we need to have similar logic in our own code at

  private static KeyManager[] keyManagers(File keyStore, String keyStorePassword)
      throws NoSuchAlgorithmException, CertificateException,
          KeyStoreException, IOException, UnrecoverableKeyException {
    KeyManagerFactory factory = KeyManagerFactory.getInstance(
      KeyManagerFactory.getDefaultAlgorithm());
    char[] passwordCharacters = keyStorePassword != null? keyStorePassword.toCharArray() : null;
    factory.init(loadKeyStore(keyStore, passwordCharacters), passwordCharacters);
    return factory.getKeyManagers();
  }

to use the key password if provided and fall back on the key store password if empty?

Copy link
Contributor Author

@hasnain-db hasnain-db Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Within Jetty, that password is read at once place:

Thanks!! This is what I had missed earlier and I wasn't sure how to pass the password.

To keep this change focused, and unblocked, though (I'll need to do more work to test this) I've filed https://issues.apache.org/jira/browse/SPARK-46132 - hope it's fine for me to put up a separate PR with just that change (+ additional tests)

.sslProvider(getSslProvider(b))
.build();
}
Expand Down Expand Up @@ -160,6 +160,7 @@ public static class Builder {
private File keyStore;
private String keyStorePassword;
private File privateKey;
private String privateKeyPassword;
private String keyPassword;
private File certChain;
private File trustStore;
Expand Down Expand Up @@ -215,16 +216,27 @@ public Builder privateKey(File privateKey) {
}

/**
* Sets the Key password
* Sets the key password
*
* @param keyPassword The password for the private key
* @param keyPassword The password for the private key in the key store
* @return The builder object
*/
public Builder keyPassword(String keyPassword) {
this.keyPassword = keyPassword;
return this;
}

/**
* Sets the private key password
*
* @param privateKeyPassword The password for the private key
* @return The builder object
*/
public Builder privateKeyPassword(String privateKeyPassword) {
this.privateKeyPassword = privateKeyPassword;
return this;
}

/**
* Sets a X.509 certificate chain file in PEM format
*
Expand Down
11 changes: 9 additions & 2 deletions common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,13 @@ public String sslRpcKeyStorePassword() {
return conf.get("spark.ssl.rpc.keyStorePassword", null);
}

/**
* The password to the private key in the key store
*/
public String sslRpcKeyPassword() {
return conf.get("spark.ssl.rpc.keyPassword", null);
}

/**
* A PKCS#8 private key file in PEM format; can be relative to the current directory
*/
Expand All @@ -314,8 +321,8 @@ public File sslRpcPrivateKey() {
/**
* The password to the private key
*/
public String sslRpcKeyPassword() {
return conf.get("spark.ssl.rpc.keyPassword", null);
public String sslRpcPrivateKeyPassword() {
return conf.get("spark.ssl.rpc.privateKeyPassword", null);
}

/**
Expand Down
22 changes: 15 additions & 7 deletions common/network-common/src/test/java/org/apache/spark/network/ssl/SslSampleConfigs.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,28 +50,35 @@ public class SslSampleConfigs {
public static final String certChainPath = getAbsolutePath("/certchain.pem");
public static final String untrustedKeyStorePath = getAbsolutePath("/untrusted-keystore");
public static final String trustStorePath = getAbsolutePath("/truststore");

public static final String unencryptedPrivateKeyPath = getAbsolutePath("/unencrypted-key.pem");
public static final String unencryptedCertChainPath =
getAbsolutePath("/unencrypted-certchain.pem");

/**
* Creates a config map containing the settings needed to enable the RPC SSL feature
* All the settings (except the enabled one) are intentionally set on the parent namespace
* so that we can verify settings inheritance works
* so that we can verify settings inheritance works. We intentionally set conflicting
* options for the key password to verify that is handled correctly.
*/
public static Map<String, String> createDefaultConfigMap() {
Map<String, String> confMap = new HashMap<String, String>();
confMap.put("spark.ssl.rpc.enabled", "true");
// Need this so the other settings get parsed
confMap.put("spark.ssl.rpc.openSslEnabled", "true");
confMap.put("spark.ssl.rpc.privateKey", SslSampleConfigs.unencryptedPrivateKeyPath);
// intentionally not set
// confMap.put("spark.ssl.rpc.privateKeyPassword", "password");
confMap.put("spark.ssl.rpc.certChain", SslSampleConfigs.unencryptedCertChainPath);
confMap.put("spark.ssl.enabled", "true");
confMap.put("spark.ssl.keyPassword", "password");
confMap.put("spark.ssl.trustStoreReloadingEnabled", "false");
confMap.put("spark.ssl.openSslEnabled", "false");
confMap.put("spark.ssl.trustStoreReloadIntervalMs", "10000");
confMap.put("spark.ssl.keyStore", SslSampleConfigs.keyStorePath);
confMap.put("spark.ssl.keyStorePassword", "password");
confMap.put("spark.ssl.privateKey", SslSampleConfigs.privateKeyPath);
confMap.put("spark.ssl.keyPassword", "password");
confMap.put("spark.ssl.certChain", SslSampleConfigs.certChainPath);
confMap.put("spark.ssl.trustStore", SslSampleConfigs.trustStorePath);
confMap.put("spark.ssl.trustStorePassword", "password");
confMap.put("spark.ssl.protocol", "TLSv1.3");
confMap.put("spark.ssl.standalone.enabled", "true");
confMap.put("spark.ssl.ui.enabled", "true");
return confMap;
}

Expand All @@ -90,6 +97,7 @@ public static Map<String, String> createDefaultConfigMapForRpcNamespace() {
confMap.put("spark.ssl.rpc.keyStorePassword", "password");
confMap.put("spark.ssl.rpc.privateKey", SslSampleConfigs.privateKeyPath);
confMap.put("spark.ssl.rpc.keyPassword", "password");
confMap.put("spark.ssl.rpc.privateKeyPassword", "password");
confMap.put("spark.ssl.rpc.certChain", SslSampleConfigs.certChainPath);
confMap.put("spark.ssl.rpc.trustStore", SslSampleConfigs.trustStorePath);
confMap.put("spark.ssl.rpc.trustStorePassword", "password");
Expand Down
21 changes: 21 additions & 0 deletions common/network-common/src/test/resources/unencrypted-certchain.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDizCCAnMCFF9A5eNs0Twi7AJJwWunO+KQRT2mMA0GCSqGSIb3DQEBCwUAMIGB
MRgwFgYDVQQDDA9EYXRhYnJpY2tzIHRlc3QxEzARBgNVBAgMCkNhbGlmb3JuaWEx
FjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xETAPBgNVBAsMCFNlcnZpY2VzMRgwFgYD
VQQKDA9EYXRhYnJpY2tzIEluYy4xCzAJBgNVBAYTAlVTMB4XDTIzMTEyMjA2MDgw
M1oXDTMzMTExOTA2MDgwM1owgYExGDAWBgNVBAMMD0RhdGFicmlja3MgdGVzdDET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzERMA8G
A1UECwwIU2VydmljZXMxGDAWBgNVBAoMD0RhdGFicmlja3MgSW5jLjELMAkGA1UE
BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAL4EfS4hPPKHwTOkQ+sNldSP
YvgLEFI8LORogSex5IHZTAzBktcaTc0W3/xS8Rd0pGOUzlDw6lLeR0M7g2pBdxsU
VZc1YDUt6R5QmBXuRco1jtPsp/Yll3LaQAk56WkiSbgPscm2QqAs1kKWd4/o9Iyt
JcwyUp7DwOVJX9Fohkayf7ktPgNZnTU0/nFaTXYwKSd2vbBs8Rq2oXlEQ88kRM3a
gUmc0y5UeFN6jt6gLNhxzJj6bZpMfojDRlW6nMMQ15Q1dps8dJWsGcOILMqQ6Deh
faS4JfAZZE6uA3b6uyNN8PalnIkJ6G+haXxsitlCprB1TF0y+gUmSPdPrZhA9nc=
-----END CERTIFICATE-----
28 changes: 28 additions & 0 deletions common/network-common/src/test/resources/unencrypted-key.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAECggEADSDhb+oe/yCdluntNBpRVjbTSKr6yqsRavX8cSrnt5Rk
eb/I/5elKnM+a1cfPwx0GJbrMqABmm5wL4qOr80FM6rBQX52tgL41sSfcmnKFjGN
RaoCQgKBPVHZVOnL3xfrDQG3WSE+5hNydYBZKjE2gOqJ8KROKC2rpbjv5AOruvX3
VS9800jq6HLmiQkjZ0eCIDlJe7f9oaDjGid2Mk1Sy/991F3pFRIuywa5OuoC7yBT
r1A2VRRQWuhXktkz8u23dMbEhU1oh2PDXDzb9gCX72c8BBZkol8YFh/64GsLmHmb
wwg1CKgPAs9R68TqXsE3RrjZ9t7iqtEIW9JDmMEGSQKBgQD4Z7ha6ukX91f8ioCm
8O9X0PjLv90VE/JFIR7Ym6YX6qrngf2iOns0utYImrYDbRhPjVBurmB+8E7LY5pS
qYjdGtT3dC6dAXrElED0siEZzORseg89I8dlWEjVa0VzUKPos3GcoLagH+mRXhhu
aXkdSaUy9jWuxwISA3MaaZvGnQKBgQDVGVUb/FI9ES7pd5RzJu+vH43EZKOMG+nF
dk2tZPU0BA4Cfjfk/GOfb7U9mWuKHRXykqPjAP6ZSK4bSAZDSGh75cApDh7WAprX
0JH5iD68Tm8KL/Lo6QAPS2/ON9hG0SX9jUdIQhsQEJgQcHTHEV0RdAKo2nsrT7tr
2F0gbgZInwKBgQChdZlozyPvRgBU0BnLaPPJWrU8iltDZhGlSV/pX1JYXVn03JNl
rSmEHqUcNqN0GqcgnjPXnVRvbfdpUDZw4G1rehNPPJ9HwjxwJgUKh/Xn9TvMHpJl
JSpn/zhoMC+WQqYnjOud6QCLl/KTYFv0+G2W0dWlCE/gaM45szBPzLFKKQKBgQCl
GV5WM1Qn2eNFoI7T9Guoe0Lj0LDhQVMJ2JFv8JME/Ms55T4628v3X53EntOxir1R
VYlBu6iFa8jwfAnWIQhKTYNmi3kah6Qd5oriEEvCquXet610A+k28FQsKhoXK71K
RyXd9tFuzdxyiB4BiRNZDU9uMO9SbBCiClyEXpnhswKBgFRwNyETdflcT3QdLSbL
FM7WWRYxum64ijMqqSPyTuvsIO8c9qwlLgqiFgiawz+MSRVX/dmhrwzBIXKDxYU1
S/pGdZO63ynOD19xSSoDh7qyPglxkGm5d8vQvmY9myUyEqqwpJHD28dBOrOyLv1K
GdxQh/QJQRFxn4SbkHG3AuiB
-----END PRIVATE KEY-----
21 changes: 21 additions & 0 deletions common/network-shuffle/src/test/resources/unencrypted-certchain.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDizCCAnMCFF9A5eNs0Twi7AJJwWunO+KQRT2mMA0GCSqGSIb3DQEBCwUAMIGB
MRgwFgYDVQQDDA9EYXRhYnJpY2tzIHRlc3QxEzARBgNVBAgMCkNhbGlmb3JuaWEx
FjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xETAPBgNVBAsMCFNlcnZpY2VzMRgwFgYD
VQQKDA9EYXRhYnJpY2tzIEluYy4xCzAJBgNVBAYTAlVTMB4XDTIzMTEyMjA2MDgw
M1oXDTMzMTExOTA2MDgwM1owgYExGDAWBgNVBAMMD0RhdGFicmlja3MgdGVzdDET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzERMA8G
A1UECwwIU2VydmljZXMxGDAWBgNVBAoMD0RhdGFicmlja3MgSW5jLjELMAkGA1UE
BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAL4EfS4hPPKHwTOkQ+sNldSP
YvgLEFI8LORogSex5IHZTAzBktcaTc0W3/xS8Rd0pGOUzlDw6lLeR0M7g2pBdxsU
VZc1YDUt6R5QmBXuRco1jtPsp/Yll3LaQAk56WkiSbgPscm2QqAs1kKWd4/o9Iyt
JcwyUp7DwOVJX9Fohkayf7ktPgNZnTU0/nFaTXYwKSd2vbBs8Rq2oXlEQ88kRM3a
gUmc0y5UeFN6jt6gLNhxzJj6bZpMfojDRlW6nMMQ15Q1dps8dJWsGcOILMqQ6Deh
faS4JfAZZE6uA3b6uyNN8PalnIkJ6G+haXxsitlCprB1TF0y+gUmSPdPrZhA9nc=
-----END CERTIFICATE-----
28 changes: 28 additions & 0 deletions common/network-shuffle/src/test/resources/unencrypted-key.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAECggEADSDhb+oe/yCdluntNBpRVjbTSKr6yqsRavX8cSrnt5Rk
eb/I/5elKnM+a1cfPwx0GJbrMqABmm5wL4qOr80FM6rBQX52tgL41sSfcmnKFjGN
RaoCQgKBPVHZVOnL3xfrDQG3WSE+5hNydYBZKjE2gOqJ8KROKC2rpbjv5AOruvX3
VS9800jq6HLmiQkjZ0eCIDlJe7f9oaDjGid2Mk1Sy/991F3pFRIuywa5OuoC7yBT
r1A2VRRQWuhXktkz8u23dMbEhU1oh2PDXDzb9gCX72c8BBZkol8YFh/64GsLmHmb
wwg1CKgPAs9R68TqXsE3RrjZ9t7iqtEIW9JDmMEGSQKBgQD4Z7ha6ukX91f8ioCm
8O9X0PjLv90VE/JFIR7Ym6YX6qrngf2iOns0utYImrYDbRhPjVBurmB+8E7LY5pS
qYjdGtT3dC6dAXrElED0siEZzORseg89I8dlWEjVa0VzUKPos3GcoLagH+mRXhhu
aXkdSaUy9jWuxwISA3MaaZvGnQKBgQDVGVUb/FI9ES7pd5RzJu+vH43EZKOMG+nF
dk2tZPU0BA4Cfjfk/GOfb7U9mWuKHRXykqPjAP6ZSK4bSAZDSGh75cApDh7WAprX
0JH5iD68Tm8KL/Lo6QAPS2/ON9hG0SX9jUdIQhsQEJgQcHTHEV0RdAKo2nsrT7tr
2F0gbgZInwKBgQChdZlozyPvRgBU0BnLaPPJWrU8iltDZhGlSV/pX1JYXVn03JNl
rSmEHqUcNqN0GqcgnjPXnVRvbfdpUDZw4G1rehNPPJ9HwjxwJgUKh/Xn9TvMHpJl
JSpn/zhoMC+WQqYnjOud6QCLl/KTYFv0+G2W0dWlCE/gaM45szBPzLFKKQKBgQCl
GV5WM1Qn2eNFoI7T9Guoe0Lj0LDhQVMJ2JFv8JME/Ms55T4628v3X53EntOxir1R
VYlBu6iFa8jwfAnWIQhKTYNmi3kah6Qd5oriEEvCquXet610A+k28FQsKhoXK71K
RyXd9tFuzdxyiB4BiRNZDU9uMO9SbBCiClyEXpnhswKBgFRwNyETdflcT3QdLSbL
FM7WWRYxum64ijMqqSPyTuvsIO8c9qwlLgqiFgiawz+MSRVX/dmhrwzBIXKDxYU1
S/pGdZO63ynOD19xSSoDh7qyPglxkGm5d8vQvmY9myUyEqqwpJHD28dBOrOyLv1K
GdxQh/QJQRFxn4SbkHG3AuiB
-----END PRIVATE KEY-----
23 changes: 18 additions & 5 deletions core/src/main/scala/org/apache/spark/SSLOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import org.apache.spark.network.util.MapConfigProvider
* @param keyStore a path to the key-store file
* @param keyStorePassword a password to access the key-store file
* @param privateKey a PKCS#8 private key file in PEM format
* @param privateKeyPassword a password to access the privateKey file
* @param keyPassword a password to access the private key in the key-store
* @param keyStoreType the type of the key-store
* @param needClientAuth set true if SSL needs client authentication
Expand Down Expand Up @@ -79,7 +80,8 @@ private[spark] case class SSLOptions(
trustStoreReloadIntervalMs: Int = 10000,
openSslEnabled: Boolean = false,
protocol: Option[String] = None,
enabledAlgorithms: Set[String] = Set.empty)
enabledAlgorithms: Set[String] = Set.empty,
privateKeyPassword: Option[String] = None)
extends Logging {

/**
Expand Down Expand Up @@ -170,6 +172,7 @@ private[spark] case class SSLOptions(
trustStorePassword.foreach(confMap.put(s"$nsp.trustStorePassword", _))
protocol.foreach(confMap.put(s"$nsp.protocol", _))
confMap.put(s"$nsp.enabledAlgorithms", enabledAlgorithms.mkString(","))
privateKeyPassword.foreach(confMap.put(s"$nsp.privateKeyPassword", _))

new MapConfigProvider(confMap)
}
Expand All @@ -178,8 +181,8 @@ private[spark] case class SSLOptions(
override def toString: String = s"SSLOptions{enabled=$enabled, port=$port, " +
s"keyStore=$keyStore, keyStorePassword=${keyStorePassword.map(_ => "xxx")}, " +
s"privateKey=$privateKey, keyPassword=${keyPassword.map(_ => "xxx")}, " +
s"keyStoreType=$keyStoreType, needClientAuth=$needClientAuth, " +
s"certChain=$certChain, trustStore=$trustStore, " +
s"privateKeyPassword=${privateKeyPassword.map(_ => "xxx")}, keyStoreType=$keyStoreType, " +
s"needClientAuth=$needClientAuth, certChain=$certChain, trustStore=$trustStore, " +
s"trustStorePassword=${trustStorePassword.map(_ => "xxx")}, " +
s"trustStoreReloadIntervalMs=$trustStoreReloadIntervalMs, " +
s"trustStoreReloadingEnabled=$trustStoreReloadingEnabled, openSSLEnabled=$openSslEnabled, " +
Expand All @@ -197,7 +200,8 @@ private[spark] object SSLOptions extends Logging {
* $ - `[ns].keyStore` - a path to the key-store file; can be relative to the current directory
* $ - `[ns].keyStorePassword` - a password to the key-store file
* $ - `[ns].privateKey` - a PKCS#8 private key file in PEM format
* $ - `[ns].keyPassword` - a password to the private key
* $ - `[ns].privateKeyPassword` - a password for the above key
* $ - `[ns].keyPassword` - a password to the private key in the key store
* $ - `[ns].keyStoreType` - the type of the key-store
* $ - `[ns].needClientAuth` - whether SSL needs client authentication
* $ - `[ns].certChain` - an X.509 certificate chain file in PEM format
Expand Down Expand Up @@ -260,6 +264,10 @@ private[spark] object SSLOptions extends Logging {
val privateKey = conf.getOption(s"$ns.privateKey").map(new File(_))
.orElse(defaults.flatMap(_.privateKey))

val privateKeyPassword = conf.getWithSubstitution(s"$ns.privateKeyPassword")
.orElse(Option(conf.getenv(ENV_RPC_SSL_PRIVATE_KEY_PASSWORD)).filter(_.trim.nonEmpty))
.orElse(defaults.flatMap(_.privateKeyPassword))

val keyPassword = conf.getWithSubstitution(s"$ns.keyPassword")
.orElse(Option(hadoopConf.getPassword(s"$ns.keyPassword")).map(new String(_)))
.orElse(Option(conf.getenv(ENV_RPC_SSL_KEY_PASSWORD)).filter(_.trim.nonEmpty))
Expand Down Expand Up @@ -320,24 +328,29 @@ private[spark] object SSLOptions extends Logging {
trustStoreReloadIntervalMs,
openSslEnabled,
protocol,
enabledAlgorithms)
enabledAlgorithms,
privateKeyPassword)
}

// Config names and environment variables for propagating SSL passwords
val SPARK_RPC_SSL_KEY_PASSWORD_CONF = "spark.ssl.rpc.keyPassword"
val SPARK_RPC_SSL_PRIVATE_KEY_PASSWORD_CONF = "spark.ssl.rpc.privateKeyPassword"
val SPARK_RPC_SSL_KEY_STORE_PASSWORD_CONF = "spark.ssl.rpc.keyStorePassword"
val SPARK_RPC_SSL_TRUST_STORE_PASSWORD_CONF = "spark.ssl.rpc.trustStorePassword"
val SPARK_RPC_SSL_PASSWORD_FIELDS: Seq[String] = Seq(
SPARK_RPC_SSL_KEY_PASSWORD_CONF,
SPARK_RPC_SSL_PRIVATE_KEY_PASSWORD_CONF,
SPARK_RPC_SSL_KEY_STORE_PASSWORD_CONF,
SPARK_RPC_SSL_TRUST_STORE_PASSWORD_CONF
)

val ENV_RPC_SSL_KEY_PASSWORD = "_SPARK_SSL_RPC_KEY_PASSWORD"
val ENV_RPC_SSL_PRIVATE_KEY_PASSWORD = "_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD"
val ENV_RPC_SSL_KEY_STORE_PASSWORD = "_SPARK_SSL_RPC_KEY_STORE_PASSWORD"
val ENV_RPC_SSL_TRUST_STORE_PASSWORD = "_SPARK_SSL_RPC_TRUST_STORE_PASSWORD"
val SPARK_RPC_SSL_PASSWORD_ENVS: Seq[String] = Seq(
ENV_RPC_SSL_KEY_PASSWORD,
ENV_RPC_SSL_PRIVATE_KEY_PASSWORD,
ENV_RPC_SSL_KEY_STORE_PASSWORD,
ENV_RPC_SSL_TRUST_STORE_PASSWORD
)
Expand Down
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/SecurityManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,8 @@ private[spark] class SecurityManager(
val map = scala.collection.mutable.Map[String, String]()
rpcSSLOptions.keyPassword.foreach(password =>
map += (SSLOptions.ENV_RPC_SSL_KEY_PASSWORD -> password))
rpcSSLOptions.privateKeyPassword.foreach(password =>
map += (SSLOptions.ENV_RPC_SSL_PRIVATE_KEY_PASSWORD -> password))
rpcSSLOptions.keyStorePassword.foreach(password =>
map += (SSLOptions.ENV_RPC_SSL_KEY_STORE_PASSWORD -> password))
rpcSSLOptions.trustStorePassword.foreach(password =>
Expand Down
21 changes: 21 additions & 0 deletions core/src/test/resources/unencrypted-certchain.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-----BEGIN CERTIFICATE-----
MIIDizCCAnMCFF9A5eNs0Twi7AJJwWunO+KQRT2mMA0GCSqGSIb3DQEBCwUAMIGB
MRgwFgYDVQQDDA9EYXRhYnJpY2tzIHRlc3QxEzARBgNVBAgMCkNhbGlmb3JuaWEx
FjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xETAPBgNVBAsMCFNlcnZpY2VzMRgwFgYD
VQQKDA9EYXRhYnJpY2tzIEluYy4xCzAJBgNVBAYTAlVTMB4XDTIzMTEyMjA2MDgw
M1oXDTMzMTExOTA2MDgwM1owgYExGDAWBgNVBAMMD0RhdGFicmlja3MgdGVzdDET
MBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzERMA8G
A1UECwwIU2VydmljZXMxGDAWBgNVBAoMD0RhdGFicmlja3MgSW5jLjELMAkGA1UE
BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAL4EfS4hPPKHwTOkQ+sNldSP
YvgLEFI8LORogSex5IHZTAzBktcaTc0W3/xS8Rd0pGOUzlDw6lLeR0M7g2pBdxsU
VZc1YDUt6R5QmBXuRco1jtPsp/Yll3LaQAk56WkiSbgPscm2QqAs1kKWd4/o9Iyt
JcwyUp7DwOVJX9Fohkayf7ktPgNZnTU0/nFaTXYwKSd2vbBs8Rq2oXlEQ88kRM3a
gUmc0y5UeFN6jt6gLNhxzJj6bZpMfojDRlW6nMMQ15Q1dps8dJWsGcOILMqQ6Deh
faS4JfAZZE6uA3b6uyNN8PalnIkJ6G+haXxsitlCprB1TF0y+gUmSPdPrZhA9nc=
-----END CERTIFICATE-----
28 changes: 28 additions & 0 deletions core/src/test/resources/unencrypted-key.pem
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDOxuEaPlFpPj6a
JgdieE5KdCA1DLPAITpjQzVpeRJNZ004jmlpzH1kH8y5pnUmzY860Upaow1BJ/eC
KxSh6YtFNiLWdErXjzfJNfgwT4TznIt8yYv3rgEYvOrxvADLA+KGibY5QOjDiyNP
uQrsRi4HE+zBE41ZgZ6h19zd8093SGNyVl7lH8gLKSKqoyAl9GaXpjrPQoMj1TIX
nMeScyCzfiX6SW2OzdcCVt8w0trSbPB19Uga9GC2DAEDp2cCt9jBeRcpZX6hqh0Z
9pCkWiwd2+MmbzaGFkutoXHjlo93cTJlonEmzvXzAMjw/qrJAf1FoYCeuW/RsTKb
MLmFSoODAgMBAAECggEADSDhb+oe/yCdluntNBpRVjbTSKr6yqsRavX8cSrnt5Rk
eb/I/5elKnM+a1cfPwx0GJbrMqABmm5wL4qOr80FM6rBQX52tgL41sSfcmnKFjGN
RaoCQgKBPVHZVOnL3xfrDQG3WSE+5hNydYBZKjE2gOqJ8KROKC2rpbjv5AOruvX3
VS9800jq6HLmiQkjZ0eCIDlJe7f9oaDjGid2Mk1Sy/991F3pFRIuywa5OuoC7yBT
r1A2VRRQWuhXktkz8u23dMbEhU1oh2PDXDzb9gCX72c8BBZkol8YFh/64GsLmHmb
wwg1CKgPAs9R68TqXsE3RrjZ9t7iqtEIW9JDmMEGSQKBgQD4Z7ha6ukX91f8ioCm
8O9X0PjLv90VE/JFIR7Ym6YX6qrngf2iOns0utYImrYDbRhPjVBurmB+8E7LY5pS
qYjdGtT3dC6dAXrElED0siEZzORseg89I8dlWEjVa0VzUKPos3GcoLagH+mRXhhu
aXkdSaUy9jWuxwISA3MaaZvGnQKBgQDVGVUb/FI9ES7pd5RzJu+vH43EZKOMG+nF
dk2tZPU0BA4Cfjfk/GOfb7U9mWuKHRXykqPjAP6ZSK4bSAZDSGh75cApDh7WAprX
0JH5iD68Tm8KL/Lo6QAPS2/ON9hG0SX9jUdIQhsQEJgQcHTHEV0RdAKo2nsrT7tr
2F0gbgZInwKBgQChdZlozyPvRgBU0BnLaPPJWrU8iltDZhGlSV/pX1JYXVn03JNl
rSmEHqUcNqN0GqcgnjPXnVRvbfdpUDZw4G1rehNPPJ9HwjxwJgUKh/Xn9TvMHpJl
JSpn/zhoMC+WQqYnjOud6QCLl/KTYFv0+G2W0dWlCE/gaM45szBPzLFKKQKBgQCl
GV5WM1Qn2eNFoI7T9Guoe0Lj0LDhQVMJ2JFv8JME/Ms55T4628v3X53EntOxir1R
VYlBu6iFa8jwfAnWIQhKTYNmi3kah6Qd5oriEEvCquXet610A+k28FQsKhoXK71K
RyXd9tFuzdxyiB4BiRNZDU9uMO9SbBCiClyEXpnhswKBgFRwNyETdflcT3QdLSbL
FM7WWRYxum64ijMqqSPyTuvsIO8c9qwlLgqiFgiawz+MSRVX/dmhrwzBIXKDxYU1
S/pGdZO63ynOD19xSSoDh7qyPglxkGm5d8vQvmY9myUyEqqwpJHD28dBOrOyLv1K
GdxQh/QJQRFxn4SbkHG3AuiB
-----END PRIVATE KEY-----
Loading