Keyed Window Buffer #32
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement a keyed window buffer - Tests for keyed window buffer - Docs for keyed window buffer Keyed window buffer works similar to system window but batches based on a key mapping, and has an optional timeout for batches. When a new key is seen, a batch is started for that key, and as new messages come in they are added to their respective keys. If a new batch is to be created, but max pending keys is reached, then the message is nacked to allow for completing the current batches. After each message, if a check mapping is supplied, it is run against the batch to determine if the batch completion needs are matched. Once a batch is complete or the specified timeout is hit, then the batch is pushed through the pipeline. On normal shutdown, any pending batches are nacked for processing upon next startup.
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Keyed window buffer works similar to system window but batches based on a key mapping, and has an optional timeout for batches. When a new key is seen, a batch is started for that key, and as new messages come in they are added to their respective keys. If a new batch is to be created, but max pending keys is reached, then the message is nacked to allow for completing the current batches. After each message, if a check mapping is supplied, it is run against the batch to determine if the batch completion needs are matched. Once a batch is complete or the specified timeout is hit, then the batch is pushed through the pipeline. On normal shutdown, any pending batches are nacked for processing upon next startup.