Skip to content

Simplify kafka client bootstrap server names and ports config #619

@jfallows

Description

@jfallows

Describe the desired outcome from the user's perspective
As a developer, I want to be able to copy and paste the bootstrap server host and port into zilla.yaml and not need to understand the details of tls server name or tcp host and port.

Acceptance criteria

  • Support configuration of server (array) option in kafka client binding for bootstrap servers
  • Eliminate need for any tls client binding options or routes to support kafka bootstrap servers
  • Eliminate need for any tcp client binding options or routes to support kafka bootstrap servers

Additional context
Today, we need to separate the components of the Kafka bootstrap server hostname and port across different client bindings to cause the correct behavior to occur at each layer in the protocol stack.

With plaintext tcp connectivity between Zilla and Kafka:

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    options:
      host: ${{env.KAFKA_HOST}}
      port: ${{env.KAFKA_PORT}}
    routes:
      - when:
          - cidr: 0.0.0.0/0

With tls encryption between Zilla and Kafka:

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    options:
      trustcacerts: true
      sni: ${{env.KAFKA_HOST}}
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    options:
      host: ${{env.KAFKA_HOST}}
      port: ${{env.KAFKA_PORT}}
    routes:
      - when:
          - cidr: 0.0.0.0/0

This can be error prone to configure, as typical Kafka client bootstrap server(s) are provided as a list of hostname:port.

Consider simplifying the configuration by adding an array of server properties to kafka binding client options.

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    routes:
      - when:
          - cidr: 0.0.0.0/0

Simplifies migration from tcp to tls.

  # Connect to Kafka
  south_kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{env.KAFKA_HOST}}:${{env.KAFKA_PORT}}
    exit: south_tls_client
  south_tls_client:
    type: tls
    kind: client
    exit: south_tcp_client
  south_tcp_client:
    type: tcp
    kind: client
    routes:
      - when:
          - cidr: 0.0.0.0/0

Note: this simplest possible approach would also require defaulting trustcacerts to true when no explicit trust option specified in tls binding, and allowing cidr of 0.0.0.0/0 by default in tcp binding.

Note: the servers list of kafka binding client options would be used to randomly select a specific server to bootstrap discovery of the brokers in the cluster.

Tasks

  • Support kafka client option for server array of bootstrap servers in the syntax host:port
  • Send (random) bootstrap server as initial proxy begin extension, same as for discovered brokers
  • Change default behavior of tcp client binding to not require host or port options, nor routes
  • Update kafka binding reference docs
  • Update tcp binding reference docs

Metadata

Metadata

Assignees

Labels

storyFeature description from user's perspective

Type

No type
No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions