[SPARK-23827] [SS] StreamingJoinExec should ensure that input data is partitioned into specific number of partitions #20941
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tdas
changed the title
Contributor
Author
brkyvz
approved these changes
Mar 29, 2018
Contributor
brkyvz
left a comment
brkyvz
left a comment
There was a problem hiding this comment.
LGTM. This is great. Left one comment
| assert(d.requiredNumPartitions.isDefined) | ||
| assert(d.requiredNumPartitions.get >= 1) | ||
| if (d != AllTuples) { | ||
| assert(d.requiredNumPartitions.get == s.stateInfo.get.numPartitions) |
Contributor
There was a problem hiding this comment.
can you also verify that this is equal to the number of partitions in the metadata?
brkyvz
reviewed
Mar 30, 2018
| assert(s.stateInfo.isDefined) | ||
| assert(s.stateInfo.get.numPartitions >= 1) | ||
| assert( | ||
| s.stateInfo.map(_.numPartitions).contains(currentStream.lastExecution.numStateStores)) |
|
Test build #88721 has finished for PR 20941 at commit
|
|
Test build #88727 has finished for PR 20941 at commit
|
gatorsmile
reviewed
Mar 30, 2018
| override def requiredChildDistribution: Seq[Distribution] = | ||
| ClusteredDistribution(leftKeys) :: ClusteredDistribution(rightKeys) :: Nil | ||
| ClusteredDistribution(leftKeys, stateInfo.map(_.numPartitions)) :: | ||
| ClusteredDistribution(rightKeys, stateInfo.map(_.numPartitions)) :: Nil |
Contributor
Author
|
Started more tests to test for flakiness. |
|
Test build #88761 has finished for PR 20941 at commit
|
|
Test build #4146 has finished for PR 20941 at commit
|
tedyu
approved these changes
Mar 30, 2018
|
Test build #4145 has finished for PR 20941 at commit
|
|
Test build #4147 has finished for PR 20941 at commit
|
|
Test build #4148 has finished for PR 20941 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Mar 30, 2018
…partitioned into specific number of partitions ## What changes were proposed in this pull request? Currently, the requiredChildDistribution does not specify the partitions. This can cause the weird corner cases where the child's distribution is `SinglePartition` which satisfies the required distribution of `ClusterDistribution(no-num-partition-requirement)`, thus eliminating the shuffle needed to repartition input data into the required number of partitions (i.e. same as state stores). That can lead to "file not found" errors on the state store delta files as the micro-batch-with-no-shuffle will not run certain tasks and therefore not generate the expected state store delta files. This PR adds the required constraint on the number of partitions. ## How was this patch tested? Modified test harness to always check that ANY stateful operator should have a constraint on the number of partitions. As part of that, the existing opt-in checks on child output partitioning were removed, as they are redundant. Author: Tathagata Das <[email protected]> Closes #20941 from tdas/SPARK-23827. (cherry picked from commit 15298b9) Signed-off-by: Tathagata Das <[email protected]>
mshtelma
pushed a commit
to mshtelma/spark
that referenced
this pull request
Apr 5, 2018
…partitioned into specific number of partitions ## What changes were proposed in this pull request? Currently, the requiredChildDistribution does not specify the partitions. This can cause the weird corner cases where the child's distribution is `SinglePartition` which satisfies the required distribution of `ClusterDistribution(no-num-partition-requirement)`, thus eliminating the shuffle needed to repartition input data into the required number of partitions (i.e. same as state stores). That can lead to "file not found" errors on the state store delta files as the micro-batch-with-no-shuffle will not run certain tasks and therefore not generate the expected state store delta files. This PR adds the required constraint on the number of partitions. ## How was this patch tested? Modified test harness to always check that ANY stateful operator should have a constraint on the number of partitions. As part of that, the existing opt-in checks on child output partitioning were removed, as they are redundant. Author: Tathagata Das <[email protected]> Closes apache#20941 from tdas/SPARK-23827.
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
…partitioned into specific number of partitions Currently, the requiredChildDistribution does not specify the partitions. This can cause the weird corner cases where the child's distribution is `SinglePartition` which satisfies the required distribution of `ClusterDistribution(no-num-partition-requirement)`, thus eliminating the shuffle needed to repartition input data into the required number of partitions (i.e. same as state stores). That can lead to "file not found" errors on the state store delta files as the micro-batch-with-no-shuffle will not run certain tasks and therefore not generate the expected state store delta files. This PR adds the required constraint on the number of partitions. Modified test harness to always check that ANY stateful operator should have a constraint on the number of partitions. As part of that, the existing opt-in checks on child output partitioning were removed, as they are redundant. Author: Tathagata Das <[email protected]> Closes apache#20941 from tdas/SPARK-23827. (cherry picked from commit 15298b9) Signed-off-by: Tathagata Das <[email protected]> Change-Id: I9dc225a765afb198e3e8719bdb3dfffd2cff95b9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, the requiredChildDistribution does not specify the partitions. This can cause the weird corner cases where the child's distribution is
SinglePartitionwhich satisfies the required distribution ofClusterDistribution(no-num-partition-requirement), thus eliminating the shuffle needed to repartition input data into the required number of partitions (i.e. same as state stores). That can lead to "file not found" errors on the state store delta files as the micro-batch-with-no-shuffle will not run certain tasks and therefore not generate the expected state store delta files.This PR adds the required constraint on the number of partitions.
How was this patch tested?
Modified test harness to always check that ANY stateful operator should have a constraint on the number of partitions. As part of that, the existing opt-in checks on child output partitioning were removed, as they are redundant.