Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error

**Describe the bug**
Trying to use zilla with Aiven Kafka I am unable to connect.

**To Reproduce**
Steps to reproduce the behavior:
1. Follow the instructions on [connecting Zilla to Aiven](https://docs.aklivity.io/zilla/latest/how-tos/connecting-to-kafka/aiven.html) and the Aiven Quick connect instructions.
2. try to start zilla and see the error


**Zilla Environment:**
Describe the Host environment including:
- Zilla start command
```
args: ["start", "-v", "-e", "-Pzilla.engine.worker.capacity=512"]
```
- Environment variables
```
env:
  ZILLA_INCUBATOR_ENABLED: "true"
  JAVA_OPTIONS: "-Dzilla.binding.kafka.debug=true -Djavax.net.debug=ssl,handshake"
```


**Attach the `zilla.yaml` config file:**
```
name: http-quickstart
vaults:
  client_vault:
    type: filesystem
    options:
      trust:
        store: /ssl/client.truststore.p12
        type: pkcs12
        password: ${{env.TRUSTSTORE_PASSWORD}}
      keys:
        store: /ssl/client.keystore.p12
        type: pkcs12
        password: ${{env.KEYSTORE_PASSWORD}}
bindings:
  # Proxy service entrypoint
  north_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port:
        - 7114
    routes:
      - when:
          - port: 7114
        exit: north_http_server
    telemetry:
      metrics:
        - stream.*
  north_http_server:
    type: http
    kind: server
    options:
      access-control:
        policy: cross-origin
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:7114
              :path: /api/*
        exit: north_rest_api_http_kafka_mapping
  # REST proxy endpoints to Kafka a topic
  north_rest_api_http_kafka_mapping:
    type: http-kafka
    kind: proxy
    routes:
      - when:
          - method: GET
            path: /api/items
        exit: north_kafka_cache_client
        with:
          capability: fetch
          topic: items-crud
          merge:
            content-type: application/json
  # Kafka sync layer
  north_kafka_cache_client:
    type: kafka
    kind: cache_client
    exit: south_kafka_cache_server
  south_kafka_cache_server:
    type: kafka
    kind: cache_server
    options:
      bootstrap:
        - items-crud
    exit: south_kafka_client
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    options:
      trust:
        - ca
      keys:
        - service_key
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
telemetry:
  exporters:
    # Enable Standard Out logs
    stdout_logs_exporter:
      type: stdout
```
**Attach the `zilla logs`:**

```
started
[http-quickstart.south_kafka_cache_server] items-crud DESCRIBE connect
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|ServerHello.java:883|Consuming ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "5C0B0FB218C89A8A3C4CE2474236447C87C2ECF94C2FC902566FED0C7A403FC5",
  "session id"          : "4D9A1F4349574653F95591B103916D08B19A6B8C136EECBA2B67AC5C8C3C1BCA",
  "cipher suite"        : "TLS_AES_256_GCM_SHA384(0x1302)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {
        "named group": x25519
        "key_exchange": {
          0000: 2D 0D 0A 05 6E 2F 93 9C   ED 94 55 91 24 C6 43 09  -...n/....U.$.C.
          0010: FC F1 3F 5F 14 50 6F 49   10 5E 19 21 C1 C1 1F 5D  ..?_.PoI.^.!...]
        }
      },
    }
  ]
}
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|SSLExtensions.java:204|Consumed extension: supported_versions
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.913 GMT|ServerHello.java:979|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.914 GMT|SSLExtensions.java:175|Ignore unsupported extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: ec_point_formats
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.915 GMT|SSLExtensions.java:175|Ignore unsupported extension: status_request_v2
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:175|Ignore unsupported extension: extended_master_secret
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:175|Ignore unsupported extension: session_ticket
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.916 GMT|SSLExtensions.java:204|Consumed extension: supported_versions
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:204|Consumed extension: key_share
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:175|Ignore unsupported extension: renegotiation_info
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|PreSharedKeyExtension.java:922|Handling pre_shared_key absence.
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.917 GMT|SSLExtensions.java:219|Ignore unavailable extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: ec_point_formats
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: status_request_v2
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: extended_master_secret
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: session_ticket
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: supported_versions
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: key_share
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: renegotiation_info
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.918 GMT|SSLExtensions.java:219|Ignore unavailable extension: pre_shared_key
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.923 GMT|SSLCipher.java:1836|KeyLimit read side: algorithm = AES/GCM/NoPadding:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.924 GMT|SSLCipher.java:1987|KeyLimit write side: algorithm = AES/GCM/NoPadding:KEYUPDATE
countdown value = 137438953472
javax.net.ssl|DEBUG|B1|engine/data#3|2024-06-28 18:56:19.925 GMT|ChangeCipherSpec.java:244|Consuming ChangeCipherSpec message
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.928 GMT|EncryptedExtensions.java:172|Consuming EncryptedExtensions handshake message (
"EncryptedExtensions": [
  "supported_groups (10)": {
    "named groups": [x25519, secp256r1, secp384r1, secp521r1, x448, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
  }
]
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:185|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:185|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:204|Consumed extension: supported_groups
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: max_fragment_length
javax.net.ssl|WARNING|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:227|Ignore impact of unsupported extension: supported_groups
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.929 GMT|SSLExtensions.java:219|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|CertificateMessage.java:1143|Consuming server Certificate handshake message (
"Certificate": {
  "certificate_request_context": "",
  "certificate_list": [  
  {
    "certificate" : {
      "version"            : "v3",
      "serial number"      : "2AA761AA17604BD87F9FB5F30D9D32FE35893F0C",
      "signature algorithm": "SHA384withRSA",
      "issuer"             : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "not before"         : "2024-06-27 13:57:41.000 GMT",
      "not  after"         : "2026-09-25 13:57:41.000 GMT",
      "subject"            : "CN=kafka-zilla-quickstart-4, O=8090602f-c2ac-493e-9a2b-a64639720a5b, ST=service",
      "subject public key" : "RSA",
      "extensions"         : [
        {
          ObjectId: 2.5.29.35 Criticality=false
          AuthorityKeyIdentifier [
          KeyIdentifier [
          0000: CD 6F A4 50 AD 16 89 3D   5D 17 0F 4C 06 5B 38 DE  .o.P...=]..L.[8.
          0010: 44 57 9D 8E                                        DW..
          ]
          ]
        },
        {
          ObjectId: 2.5.29.19 Criticality=false
          BasicConstraints:[
            CA:false
            PathLen: undefined
          ]
        },
        {
          ObjectId: 2.5.29.15 Criticality=false
          KeyUsage [
            DigitalSignature
            Key_Encipherment
          ]
        },
        {
          ObjectId: 2.5.29.17 Criticality=false
          SubjectAlternativeName [
            DNSName: 10.162.0.71
            DNSName: public-kafka-zilla-quickstart-zilla-quickstart.l.aivencloud.com
            DNSName: public-n-kafka-zilla-quickstart-4.l.aivencloud.com
            IPAddress: fda7:a938:5bfe:5fa6:0:4c2:5e57:1c42
            IPAddress: 34.47.20.235
            DNSName: kafka-zilla-quickstart-zilla-quickstart.l.aivencloud.com
            IPAddress: 2600:1900:40e0:504:0:47:0:0
            DNSName: 34.47.20.235
            DNSName: n-kafka-zilla-quickstart-4.l.aivencloud.com
            DNSName: fda7:a938:5bfe:5fa6:0:4c2:5e57:1c42
            IPAddress: 10.162.0.71
            DNSName: *.l.aivencloud.com
            DNSName: 2600:1900:40e0:504:0:47::
          ]
        },
        {
          ObjectId: 2.5.29.14 Criticality=false
          SubjectKeyIdentifier [
          KeyIdentifier [
          0000: 0F 28 84 1A C3 4A 42 95   0F 2E 6A 83 C8 79 3E D0  .(...JB...j..y>.
          0010: 61 9C 00 7E                                        a...
          ]
          ]
        }
      ]}
    "extensions": {
      <no extension>
    }
  },
  {
    "certificate" : {
      "version"            : "v3",
      "serial number"      : "0DD54456AF9909A495C584F56169D76C3DC2D319",
      "signature algorithm": "SHA384withRSA",
      "issuer"             : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "not before"         : "2024-06-26 21:17:00.000 GMT",
      "not  after"         : "2034-06-24 21:17:00.000 GMT",
      "subject"            : "CN=8090602f-c2ac-493e-9a2b-a64639720a5b Project CA",
      "subject public key" : "RSA",
      "extensions"         : [
        {
          ObjectId: 2.5.29.19 Criticality=false
          BasicConstraints:[
            CA:true
            PathLen:0
          ]
        },
        {
          ObjectId: 2.5.29.15 Criticality=false
          KeyUsage [
            Key_CertSign
            Crl_Sign
          ]
        },
        {
          ObjectId: 2.5.29.14 Criticality=false
          SubjectKeyIdentifier [
          KeyIdentifier [
          0000: CD 6F A4 50 AD 16 89 3D   5D 17 0F 4C 06 5B 38 DE  .o.P...=]..L.[8.
          0010: 44 57 9D 8E                                        DW..
          ]
          ]
        }
      ]}
    "extensions": {
      <no extension>
    }
  },
]
}
)
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|SSLExtensions.java:185|Ignore unavailable extension: status_request
javax.net.ssl|DEBUG|32|engine/task#0|2024-06-28 18:56:19.934 GMT|SSLExtensions.java:185|Ignore unavailable extension: status_request
javax.net.ssl|ERROR|32|engine/task#0|2024-06-28 18:56:19.936 GMT|TransportContext.java:370|Fatal (INTERNAL_ERROR): Unhandled exception (
"throwable" : {
  java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:97)
    at java.base/sun.security.validator.Validator.getInstance(Validator.java:173)
    at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:308)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:183)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:254)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1304)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1203)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1146)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1274)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
    at io.aklivity.zilla.runtime.engine@0.9.83/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker$EngineSignaler.invokeAndSignal(EngineWorker.java:2063)
    at io.aklivity.zilla.runtime.engine@0.9.83/io.aklivity.zilla.runtime.engine.internal.registry.EngineWorker$EngineSignaler.lambda$signalTask$3(EngineWorker.java:1983)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1570)
  Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
    at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
    at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
    at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:94)
    ... 21 more}

)

```

**Kafka Environment:**

- Provider: Aiven
- Version: 3.7
- Config: Client cert
![image](https://github.com/aklivity/zilla/assets/3012311/d1cfc93b-f999-47e5-b5c5-396284515f47)


**Client Environment:**

the `/opt/bitnami/kafka/bin/kafka-topics.sh --list` command works with the below SSL configs:
```
security.protocol=SSL
ssl.protocol=TLS
ssl.keystore.type=PKCS12
ssl.keystore.location=client.keystore.p12
ssl.keystore.password=pass
ssl.key.password=pass
ssl.truststore.location=client.truststore.jks
ssl.truststore.password=pass
ssl.truststore.type=JKS
```

**Additional context**
Aiven Quick connect instructions:
3. Create keystore and truststore:
3.1 Use the openssl utility to create the keystore with the service.key and service.cert files downloaded previously:

```
openssl pkcs12 -export -inkey service.key -in service.cert -out client.keystore.p12 -name service_key
```

Enter a password to protect the keystore and the key, when prompted.
3.2 In the folder where the certificates are stored, use the keytool utility to create the truststore with the ca.pem file as input:

```
keytool -import -file ca.pem -alias CA -keystore client.truststore.jks
```

Enter a password to protect the trust stores and reply yes to confirm trusting the CA certificate, when prompted.
The result are the keystore named client.keystore.p12 and truststore named client.truststore.jks that can be used for client applications configuration in the next steps.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Connecting to Aiven Kafka over TLS Throws an java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty Error #1115

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Connecting to Aiven Kafka over TLS Throws an `java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty` Error #1115