Describe the desired outcome from the user's perspective
As a developer, I want to be able to copy and paste the bootstrap server host and port into zilla.yaml and not need to understand the details of tls server name or tcp host and port.
Acceptance criteria
- Support configuration of
server (array) option in kafka client binding for bootstrap servers
- Eliminate need for any
tls client binding options or routes to support kafka bootstrap servers
- Eliminate need for any
tcp client binding options or routes to support kafka bootstrap servers
Additional context
Today, we need to separate the components of the Kafka bootstrap server hostname and port across different client bindings to cause the correct behavior to occur at each layer in the protocol stack.
With plaintext tcp connectivity between Zilla and Kafka:
# Connect to Kafka
south_kafka_client:
type: kafka
kind: client
exit: south_tcp_client
south_tcp_client:
type: tcp
kind: client
options:
host: ${{env.KAFKA_HOST}}
port: ${{env.KAFKA_PORT}}
routes:
- when:
- cidr: 0.0.0.0/0
With tls encryption between Zilla and Kafka:
# Connect to Kafka
south_kafka_client:
type: kafka
kind: client
exit: south_tls_client
south_tls_client:
type: tls
kind: client
options:
trustcacerts: true
sni: ${{env.KAFKA_HOST}}
exit: south_tcp_client
south_tcp_client:
type: tcp
kind: client
options:
host: ${{env.KAFKA_HOST}}
port: ${{env.KAFKA_PORT}}
routes:
- when:
- cidr: 0.0.0.0/0
This can be error prone to configure, as typical Kafka client bootstrap server(s) are provided as a list of hostname:port.
Consider simplifying the configuration by adding an array of server properties to kafka binding client options.
# Connect to Kafka
south_kafka_client:
type: kafka
kind: client
options:
servers:
- ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
exit: south_tcp_client
south_tcp_client:
type: tcp
kind: client
routes:
- when:
- cidr: 0.0.0.0/0
Simplifies migration from tcp to tls.
# Connect to Kafka
south_kafka_client:
type: kafka
kind: client
options:
servers:
- ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
exit: south_tls_client
south_tls_client:
type: tls
kind: client
exit: south_tcp_client
south_tcp_client:
type: tcp
kind: client
routes:
- when:
- cidr: 0.0.0.0/0
Note: this simplest possible approach would also require defaulting trustcacerts to true when no explicit trust option specified in tls binding, and allowing cidr of 0.0.0.0/0 by default in tcp binding.
Note: the servers list of kafka binding client options would be used to randomly select a specific server to bootstrap discovery of the brokers in the cluster.
Tasks
Describe the desired outcome from the user's perspective
As a developer, I want to be able to copy and paste the bootstrap server host and port into
zilla.yamland not need to understand the details oftlsserver name ortcphost and port.Acceptance criteria
server(array) option inkafkaclientbinding for bootstrap serverstlsclientbinding options or routes to supportkafkabootstrap serverstcpclientbinding options or routes to supportkafkabootstrap serversAdditional context
Today, we need to separate the components of the Kafka bootstrap server hostname and port across different client bindings to cause the correct behavior to occur at each layer in the protocol stack.
With plaintext
tcpconnectivity between Zilla and Kafka:With
tlsencryption between Zilla and Kafka:This can be error prone to configure, as typical Kafka client bootstrap server(s) are provided as a list of
hostname:port.Consider simplifying the configuration by adding an array of
serverproperties tokafkabindingclientoptions.Simplifies migration from
tcptotls.Note: this simplest possible approach would also require defaulting
trustcacertstotruewhen no explicittrustoption specified intlsbinding, and allowingcidrof0.0.0.0/0by default intcpbinding.Note: the
serverslist ofkafkabindingclientoptions would be used to randomly select a specific server to bootstrap discovery of the brokers in the cluster.Tasks
kafkaclientoption forserverarray of bootstrap servers in the syntaxhost:portproxybeginextension, same as for discovered brokerstcpclientbinding to not requirehostorportoptions, norrouteskafkabinding reference docstcpbinding reference docs