Mqtt-kafka single group support cont#731
Conversation
jfallows
left a comment
jfallows
left a comment
There was a problem hiding this comment.
We also need scenario(s) to check the behavior when producer id changes on the same client produce stream, either to a different producer id value or to no producer id.
This will happen frequently if multiple clients are producing at a mixture of qos to the same kafka topic such as mqtt-messages.
| if (client.encodeSlot != NO_SLOT && | ||
| maxEncodeableBytes > encodePool.slotCapacity()) | ||
| (maxEncodeableBytes > encodePool.slotCapacity() || | ||
| client.producerId != producerId && client.producerEpoch != producerEpoch)) |
There was a problem hiding this comment.
| client.producerId != producerId && client.producerEpoch != producerEpoch)) | |
| client.producerId != producerId || client.producerEpoch != producerEpoch)) |
only need one of these to differ to trigger request since both are expected to be consistent for entire batch, right?
| .producerEpoch(client.producerEpoch) | ||
| .baseSequence(sequence) | ||
| .recordCount(encodeableRecordCount) | ||
| .build(); |
There was a problem hiding this comment.
Need to reset producerId and producerEpoch after encoding request so that next request can batch as expected?
| } | ||
|
|
||
| if (client.producerId == RECORD_BATCH_PRODUCER_ID_NONE || | ||
| client.producerId != producerId) |
There was a problem hiding this comment.
Should this be && instead?
Please remove extra whitespace around !=.
| if (client.producerId == RECORD_BATCH_PRODUCER_ID_NONE || | ||
| client.producerId != producerId) | ||
| { | ||
| client.sequence = sequence; |
There was a problem hiding this comment.
Suggest we rename to client.baseSequence.
| .typeId(zilla:id("kafka")) | ||
| .produce() | ||
| .topic("test") | ||
| .partition(1) |
There was a problem hiding this comment.
We need to match producerId and producerEpoch here too, right?
There was a problem hiding this comment.
it is part of dataEx I have filled missing ones. We can't send producerId and producerEpoch as part of beginEx since we don't know the partitions and we need partitions to fetch.
| .typeId(zilla:id("kafka")) | ||
| .produce() | ||
| .topic("test") | ||
| .partition(0) |
There was a problem hiding this comment.
producerId and producerEpoch?
jfallows
left a comment
jfallows
left a comment
There was a problem hiding this comment.
Looks pretty good.
Please add a negative test to reject gap in sequence for non-zero producerId in client produce factory.
Please add a test for replaying sequence from earlier value, but still no gaps.
| .merged() | ||
| .produce() | ||
| .key("a") | ||
| .hashKey("key7") |
There was a problem hiding this comment.
Does produce flush need both key and hashKey or just hashKey?
2 participants
Description
Mqtt-kafka single group support cont
Fixes #698