Add support for kafka producer config linger.ms in kafka sink

[punit-kulal]

linger.ms defines the maximum interval for buffering messages in the client before being produced to kafka. This improves efficiency of the producer and also greatly reduces the load on kafka.

The interval and buffering also ensures better compression when enabled.

More Reading: https://github.com/confluentinc/librdkafka/blob/master/INTRODUCTION.md#performance

Open Questions:

• Other configuration related to kafka sink are similar to flags, would you want this configuration be also similar to switch where the recommended config(1000) is behind a flag and in the code or should it be similar to the min_commit interval config we have for firehose.

Todo:

• Update version

[lavkesh]

lgtm

some changes required

[sumitaich1998]

just add "in milliseconds" after "max interval"

[punit-kulal]

Added.

[sumitaich1998]

also, shouldn't we also add the batch.size config for the kafka producer along with linger.ms?

[punit-kulal]

also, shouldn't we also add the batch.size config for the kafka producer along with linger.ms?

The default batch.size works well for most cases. Reason for custom batch.size config would be ?

[sumitaich1998]

also, shouldn't we also add the batch.size config for the kafka producer along with linger.ms?

The default batch.size works well for most cases. Reason for custom batch.size config would be ?

just so that we can tune it later for a particular dagger if we want to, we can just keep the default value in odin, we wont expose it in UI

[punit-kulal]

also, shouldn't we also add the batch.size config for the kafka producer along with linger.ms?

The default batch.size works well for most cases. Reason for custom batch.size config would be ?

just so that we can tune it later for a particular dagger if we want to, we can just keep the default value in odin, we wont expose it in UI

That can be out of the scope of the PR and can be picked independently. The scope of this PR is to expose this certain config and this helps with efficient produce of messages. There are a multiple other producer configs present as well which are currently not exposed. And can be optionally exposed in a future PR. WDYT ?

[sumitaich1998]

also, shouldn't we also add the batch.size config for the kafka producer along with linger.ms?

The default batch.size works well for most cases. Reason for custom batch.size config would be ?

just so that we can tune it later for a particular dagger if we want to, we can just keep the default value in odin, we wont expose it in UI

That can be out of the scope of the PR and can be picked independently. The scope of this PR is to expose this certain config and this helps with efficient produce of messages. There are a multiple other producer configs present as well which are currently not exposed. And can be optionally exposed in a future PR. WDYT ?

yeah ok should be fine

[sumitaich1998]

lgtm