Skip to content

[SPARK-23585][SQL] Add interpreted execution to UnwrapOption #20736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mgaido91 wants to merge 3 commits into from
Closed

[SPARK-23585][SQL] Add interpreted execution to UnwrapOption #20736

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
804394b
[SPARK-23586][SQL] Add interpreted execution to UnwrapOption
mgaido91 Mar 5, 2018
d899f44
use checkEvaluation
mgaido91 Mar 5, 2018
fc202c6
fix jira nunber
mgaido91 Mar 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[SPARK-23586][SQL] Add interpreted execution to UnwrapOption
  • Loading branch information
@mgaido91
mgaido91 committed Mar 5, 2018
commit 804394b44e65142c0af3eb3dc19bab29ed13875d
10 changes: 8 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,8 +382,14 @@ case class UnwrapOption(

override def inputTypes: Seq[AbstractDataType] = ObjectType :: Nil

override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")
override def eval(input: InternalRow): Any = {
val inputObject = child.eval(input)
Copy link
Contributor

@hvanhovell hvanhovell Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: you can also use pattern matching here. That saves you some typing:

child.eval(input) match {
  case Some(value) => value
  case _ => null
}

Copy link
Contributor Author

@mgaido91 mgaido91 Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your comment. I haven't used pattern matching because it is a bit slower than using if...else and I'd prefer the fastest way here. What do you think?

Copy link
Contributor

@hvanhovell hvanhovell Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine either way. This is going to be slow anyway. Let's leave as it is.

if (inputObject == null) {
null
} else {
inputObject.asInstanceOf[Option[_]].orNull
}
}

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val javaType = CodeGenerator.javaType(dataType)
Expand Down
14 changes: 13 additions & 1 deletion ...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.expressions
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.catalyst.expressions.objects.Invoke
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, UnwrapOption}
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData}
import org.apache.spark.sql.types.{IntegerType, ObjectType}

Expand Down Expand Up @@ -66,4 +66,16 @@ class ObjectExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
checkEvalutionWithUnsafeProjection(
mapEncoder.serializer.head, mapExpected, mapInputRow)
}

test("SPARK-23586: UnwrapOption should support interpreted execution") {
Copy link
Contributor

@hvanhovell hvanhovell Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just found out that there are literally no unit tests for this expression :(...

Can you use checkEvaluation? This will make sure both code gen and interpreted mode are tested?

val inputRowWithSome = InternalRow.fromSeq(Seq(Some(1)))
val inputRowWithNone = InternalRow.fromSeq(Seq(None))
val inputRowWithNull = InternalRow.fromSeq(Seq(null))
val cls = classOf[Option[Int]]
val inputObject = BoundReference(0, ObjectType(cls), nullable = true)
val unwrapObject = UnwrapOption(IntegerType, inputObject)
assert(unwrapObject.eval(inputRowWithSome) == 1)
assert(unwrapObject.eval(inputRowWithNone) == null)
assert(unwrapObject.eval(inputRowWithNull) == null)
}
}