[SPARK-23976][Core] Detect length overflow in UTF8String.concat()/ByteArray.concat() #21064
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
Test build #89338 has finished for PR 21064 at commit
|
Member
Author
|
ping @hvanhovell |
hvanhovell
approved these changes
Apr 19, 2018
Contributor
|
@hvanhovell seems this accidentally not get merged? |
Member
Author
|
ping @hvanhovell |
1 similar comment
Member
Author
|
ping @hvanhovell |
hvanhovell
approved these changes
May 2, 2018
Contributor
hvanhovell
left a comment
hvanhovell
left a comment
There was a problem hiding this comment.
LGTM - merging to master. Thanks!
Contributor
|
@kiszk sorry about the delay. |
Contributor
|
shall we backport it to 2.3? |
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Dec 6, 2018
…eArray.concat() apache#21064 This PR detects length overflow if total elements in inputs are not acceptable. For example, when the three inputs has 0x7FFF_FF00, 0x7FFF_FF00, and 0xE00, we should detect length overflow since we cannot allocate such a large structure on byte[]. On the other hand, the current algorithm can allocate the result structure with 0x1000-byte length due to integer sum overflow.
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Sep 20, 2019
…)/ByteArray.concat() apache#21064 This PR detects length overflow if total elements in inputs are not acceptable. For example, when the three inputs has 0x7FFF_FF00, 0x7FFF_FF00, and 0xE00, we should detect length overflow since we cannot allocate such a large structure on byte[]. On the other hand, the current algorithm can allocate the result structure with 0x1000-byte length due to integer sum overflow.
MaxGekk
pushed a commit
that referenced
this pull request
Apr 12, 2021
…erflow ### What changes were proposed in this pull request? Add check if the byte length over `int`. ### Why are the changes needed? We encounter a very extreme case with expression `concat_ws`, and the error msg is ``` Caused by: java.lang.NegativeArraySizeException at org.apache.spark.unsafe.types.UTF8String.concatWs ``` Seems the `UTF8String.concat` has already done the length check at [#21064](#21064), so it's better to add in `concatWs`. ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? It's too heavy to add the test. Closes #32106 from ulysses-you/SPARK-35005. Authored-by: ulysses-you <[email protected]> Signed-off-by: Max Gekk <[email protected]>
a0x8o
added a commit
to a0x8o/spark
that referenced
this pull request
Apr 12, 2021
…erflow ### What changes were proposed in this pull request? Add check if the byte length over `int`. ### Why are the changes needed? We encounter a very extreme case with expression `concat_ws`, and the error msg is ``` Caused by: java.lang.NegativeArraySizeException at org.apache.spark.unsafe.types.UTF8String.concatWs ``` Seems the `UTF8String.concat` has already done the length check at [#21064](apache/spark#21064), so it's better to add in `concatWs`. ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? It's too heavy to add the test. Closes #32106 from ulysses-you/SPARK-35005. Authored-by: ulysses-you <[email protected]> Signed-off-by: Max Gekk <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR detects length overflow if total elements in inputs are not acceptable.
For example, when the three inputs has
0x7FFF_FF00,0x7FFF_FF00, and0xE00, we should detect length overflow since we cannot allocate such a large structure onbyte[].On the other hand, the current algorithm can allocate the result structure with
0x1000-byte length due to integer sum overflow.How was this patch tested?
Existing UTs.
If we would create UTs, we need large heap (6-8GB). It may make test environment unstable.
If it is necessary to create UTs, I will create them.