Skip to content

[SPARK-35152][SQL] ANSI mode: IntegralDivide throws exception on overflow #32260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gengliangwang wants to merge 5 commits into from
Closed

[SPARK-35152][SQL] ANSI mode: IntegralDivide throws exception on overflow #32260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1658f5b
throw error on corner case
gengliangwang Apr 20, 2021
d91fa4e
address comment
gengliangwang Apr 20, 2021
9ade940
fix
gengliangwang Apr 20, 2021
a6ad9bf
refactor
gengliangwang Apr 20, 2021
64f1431
revise
gengliangwang Apr 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,9 @@ trait DivModLike extends BinaryArithmetic {

protected def decimalToDataTypeCodeGen(decimalResult: String): String = decimalResult

// Whether we should check overflow or not in ANSI mode.
protected def checkDivideOverflow: Boolean = false
Copy link
Contributor

@cloud-fan cloud-fan Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this simply left.dataType == LongType? For Divide, the child is either double or decimal, so won't hit this branch.

Under the hood, what we really want to handle is Long.MinValue / -1, so check left.dataType == LongType looks reasonable.

Copy link
Member Author

@gengliangwang gengliangwang Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, Remainder's left can be Long as well


override def nullable: Boolean = true

private lazy val isZero: Any => Boolean = right.dataType match {
Expand All @@ -425,6 +428,9 @@ trait DivModLike extends BinaryArithmetic {
// when we reach here, failOnError must bet true.
throw QueryExecutionErrors.divideByZeroError
}
if (checkDivideOverflow && input1 == Long.MinValue && input2 == -1) {
throw QueryExecutionErrors.overflowInIntegralDivideError()
}
evalOperation(input1, input2)
}
}
Expand All @@ -450,6 +456,15 @@ trait DivModLike extends BinaryArithmetic {
} else {
s"($javaType)(${eval1.value} $symbol ${eval2.value})"
}
val checkIntegralDivideOverflow = if (checkDivideOverflow) {
s"""
|if (${eval1.value} == ${Long.MinValue}L && ${eval2.value} == -1)
| throw QueryExecutionErrors.overflowInIntegralDivideError();
|""".stripMargin
} else {
""
}

// evaluate right first as we have a chance to skip left if right is 0
if (!left.nullable && !right.nullable) {
val divByZero = if (failOnError) {
Expand All @@ -465,6 +480,7 @@ trait DivModLike extends BinaryArithmetic {
$divByZero
} else {
${eval1.code}
$checkIntegralDivideOverflow
${ev.value} = $operation;
}""")
} else {
Expand All @@ -486,6 +502,7 @@ trait DivModLike extends BinaryArithmetic {
${ev.isNull} = true;
} else {
$failOnErrorBranch
$checkIntegralDivideOverflow
${ev.value} = $operation;
}
}""")
Expand Down Expand Up @@ -546,6 +563,11 @@ case class IntegralDivide(

def this(left: Expression, right: Expression) = this(left, right, SQLConf.get.ansiEnabled)

override def checkDivideOverflow: Boolean = left.dataType match {
case LongType if failOnError => true
case _ => false
}

override def inputType: AbstractDataType = TypeCollection(LongType, DecimalType)

override def dataType: DataType = LongType
Expand Down
4 changes: 4 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ object QueryExecutionErrors {
new ArithmeticException("Overflow in sum of decimals.")
}

def overflowInIntegralDivideError(): ArithmeticException = {
new ArithmeticException("Overflow in integral divide.")
}

def mapSizeExceedArraySizeWhenZipMapError(size: Int): RuntimeException = {
new RuntimeException(s"Unsuccessful try to zip maps with $size " +
"unique keys due to exceeding the array size limit " +
Expand Down
7 changes: 7 additions & 0 deletions .../src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,13 @@ class ArithmeticExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
}
}

test("IntegralDivide: throw exception on overflow under ANSI mode") {
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
checkExceptionInExpression[ArithmeticException](
IntegralDivide(Literal(Long.MinValue), Literal(-1L)), "Overflow in integral divide.")
}
}

test("% (Remainder)") {
testNumericDataTypes { convert =>
val left = Literal(convert(1))
Expand Down