[SPARK-36717][CORE] Incorrect order of variable initialization may lead incorrect behavior #33957
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ad to incorrect behavior
Contributor
|
Ok to test |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
mridulm
reviewed
Sep 10, 2021
Contributor
mridulm
left a comment
mridulm
left a comment
There was a problem hiding this comment.
The fix looks good to me.
Can you add a test to surface the issue (without the fix) and verify it is addressed with this fix.
|
Test build #143153 has finished for PR 33957 at commit
|
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1 for @mridulm 's comment. It would be great if we can have a small test case.
Member
|
BTW, thank you for making a PR, @daugraph . |
HyukjinKwon
approved these changes
Sep 12, 2021
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
same comments as @dongjoon-hyun and @mridulm. Looks good otherwise.
srowen
approved these changes
Sep 12, 2021
mridulm
reviewed
Sep 26, 2021
mridulm
approved these changes
Sep 28, 2021
Author
|
I have no idea about how to add the test case, plz give me some advice, thanks |
srowen
reviewed
Oct 3, 2021
Member
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
I wonder if you can simply add a test alongside other broadcast tests, that sets the flag here to true in conf, then instantiates this object, then checks whether the flag is true? it would only work if this fix works
Contributor
|
You could create a broadcast variable and check if |
Member
|
If we're stuck on adding a trivial test here, I'm inclined to just proceed - it's a clean fix and existing tests pass |
Member
|
Jenkins test this please |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #144000 has finished for PR 33957 at commit
|
Member
|
Yeah, I am okie too. |
srowen
pushed a commit
that referenced
this pull request
Oct 8, 2021
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes #33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
srowen
pushed a commit
that referenced
this pull request
Oct 8, 2021
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes #33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
srowen
pushed a commit
that referenced
this pull request
Oct 8, 2021
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes #33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
Member
|
Merged to master/3.2/3.1/3.0 |
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Dec 8, 2021
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes apache#33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
fishcus
pushed a commit
to fishcus/spark
that referenced
this pull request
Jan 12, 2022
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes apache#33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
catalinii
pushed a commit
to lyft/spark
that referenced
this pull request
Feb 22, 2022
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes apache#33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
catalinii
pushed a commit
to lyft/spark
that referenced
this pull request
Mar 4, 2022
…ad incorrect behavior
### What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
### Why are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
Closes apache#33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 65f6a7c)
Signed-off-by: Sean Owen <[email protected]>
wangyum
pushed a commit
that referenced
this pull request
May 26, 2023
…ad incorrect behavior
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
```scala
class Broadcast {
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
var checksumEnabled = false
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
false
```
Snippet 2
```scala
class Broadcast {
var checksumEnabled = false
def setConf(): Unit = {
checksumEnabled = true
}
setConf()
}
println(new Broadcast().checksumEnabled)
```
output:
```scala
true
```
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
No
No
Closes #33957 from daugraph/branch0.
Authored-by: daugraph <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Incorrect order of variable initialization may lead to incorrect behavior, related code: TorrentBroadcast.scala , TorrentBroadCast will get wrong checksumEnabled value after initialization, this may not be what we need, we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Supplement:
Snippet 1
output:
falseSnippet 2
output:
trueWhy are the changes needed?
we can move L94 front of setConf(SparkEnv.get.conf) to avoid this.
Does this PR introduce any user-facing change?
No
How was this patch tested?
No