Skip to content

Conversation

@punit-kulal
Copy link

linger.ms defines the maximum interval for buffering messages in the client before being produced to kafka. This improves efficiency of the producer and also greatly reduces the load on kafka.

The interval and buffering also ensures better compression when enabled.

More Reading: https://github.com/confluentinc/librdkafka/blob/master/INTRODUCTION.md#performance

Open Questions:

  • Other configuration related to kafka sink are similar to flags, would you want this configuration be also similar to switch where the recommended config(1000) is behind a flag and in the code or should it be similar to the min_commit interval config we have for firehose.

Todo:

  • Add documentation for the config exposed[Waitiing on finalisation for config exposure pattern]
  • Update version

Description: Add three constants to support linger ms. The default value set is 5 which is the default for the kafka producer client.
Description: Integrate configuration support for linger ms in the kafka producer initialisation.
Description: The default of 5 ms is for librd kafka clients. Java client has default as 0. Hence, updating it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant