[SPARK-24038][SS] Refactor continuous writing to its own class #21116
Conversation
|
Test build #89654 has finished for PR 21116 at commit
|
xuanyuanking
left a comment
xuanyuanking
left a comment
There was a problem hiding this comment.
Cool refactor, more clear logic of CP plans, just a nit.
| val epochCoordinator = EpochCoordinatorRef.get( | ||
| context.getLocalProperty(ContinuousExecution.EPOCH_COORDINATOR_ID_KEY), | ||
| SparkEnv.get) | ||
| val currentMsg: WriterCommitMessage = null |
There was a problem hiding this comment.
currentMsg is no longer needed?
There was a problem hiding this comment.
Good point. I see its no long used anywhere. That raises two questions.
- This should be removed.
- Does this continuous code path not have to send back a WriterMessage? How is that working?
There was a problem hiding this comment.
Nvm. I see the msg being sent back using the epochCoordinator. Then lets just remove the currentMsg
|
|
| } | ||
|
|
||
| val rdd = query.execute() | ||
| val messages = new Array[WriterCommitMessage](rdd.partitions.length) |
There was a problem hiding this comment.
Is this really needed. The only use of it is in the logInfo before, that too, only in the length, which is effectively rdd.partitions.length.
|
addressed comments |
|
Test build #89747 has finished for PR 21116 at commit
|
Refactor continuous writing to its own class. See WIP jose-torres#13 for the overall direction this is going, but I think this PR is very isolated and necessary anyway. existing unit tests - refactoring only Author: Jose Torres <[email protected]> Closes apache#21116 from jose-torres/SPARK-24038. Ref: LIHADOOP-48531 RB=1850759 G=superfriends-reviewers R=zolin,fli,latang,mshen,yezhou A=
4 participants
What changes were proposed in this pull request?
Refactor continuous writing to its own class.
See WIP jose-torres#13 for the overall direction this is going, but I think this PR is very isolated and necessary anyway.
How was this patch tested?
existing unit tests - refactoring only