Skip to content

Add support for kafka producer config linger.ms in kafka sink #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sumitaich1998 merged 10 commits into from
May 3, 2023
Merged

Add support for kafka producer config linger.ms in kafka sink #7

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3e37a87
feat: add configuration to support linger ms for kafka sink
punit-kulal Mar 11, 2023
a913011
feat: support for linger ms configuration in kafka producer
punit-kulal Mar 11, 2023
fc7260a
fix: change default linger ms value to 0.
punit-kulal Mar 11, 2023
20b3d2f
fix: Fix import reference of constants
punit-kulal Mar 21, 2023
8f10ea2
feat: add documentation for the config for linger ms.
punit-kulal Mar 27, 2023
e1b0079
feat: bump version.txt
punit-kulal Mar 28, 2023
ece4d01
feat: modify test to verify linger ms being set in kakfa producer con…
punit-kulal Apr 22, 2023
e2e6d0f
feat: added validateion and test to check correctness of linger ms co…
punit-kulal Apr 22, 2023
de87983
doc: added in milliseconds in the document
punit-kulal Apr 22, 2023
b964802
fix: added more information to the error for linger ms check
punit-kulal Apr 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
feat: add configuration to support linger ms for kafka sink 
Description: Add three constants to support linger ms. The default value set is 5 which is the default for the kafka producer client.
  • Loading branch information
@punit-kulal
punit-kulal committed Mar 28, 2023
commit 3e37a87a5d21d4c5a19b8b3e4e14624811d1b5a6
3 changes: 3 additions & 0 deletions dagger-core/src/main/java/com/gotocompany/dagger/core/utils/Constants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,14 @@ public class Constants {
public static final String SINK_KAFKA_JSON_SCHEMA_KEY = "SINK_KAFKA_JSON_SCHEMA";
public static final String SINK_KAFKA_DATA_TYPE = "SINK_KAFKA_DATA_TYPE";
public static final String SINK_KAFKA_PRODUCE_LARGE_MESSAGE_ENABLE_KEY = "SINK_KAFKA_PRODUCE_LARGE_MESSAGE_ENABLE";
public static final String SINK_KAFKA_LINGER_MS_KEY = "SINK_KAFKA_LINGER_MS";
public static final boolean SINK_KAFKA_PRODUCE_LARGE_MESSAGE_ENABLE_DEFAULT = false;
public static final String SINK_KAFKA_COMPRESSION_TYPE_KEY = "compression.type";
public static final String SINK_KAFKA_LINGER_MS_CONFIG_KEY = "linger.ms";
public static final String SINK_KAFKA_COMPRESSION_TYPE_DEFAULT = "snappy";
public static final String SINK_KAFKA_MAX_REQUEST_SIZE_KEY = "max.request.size";
public static final String SINK_KAFKA_MAX_REQUEST_SIZE_DEFAULT = "20971520";
public static final String SINK_KAFKA_LINGER_MS_DEFAULT = "5";

public static final String ES_TYPE = "ES";
public static final String HTTP_TYPE = "HTTP";
Expand Down