Skip to content

[SPARK-22856][SQL] Add wrappers for codegen output and nullability #20043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 14 commits into from
Closed

[SPARK-22856][SQL] Add wrappers for codegen output and nullability #20043

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
5ace8b8
Add wrappers for codegen output.
viirya Dec 21, 2017
d120750
Merge remote-tracking branch 'upstream/master' into SPARK-22856
viirya Dec 21, 2017
81c9b6e
Fix equality check with string.
viirya Dec 21, 2017
53926cc
Address comments.
viirya Dec 21, 2017
69422d4
Merge remote-tracking branch 'upstream/master' into SPARK-22856
viirya Dec 22, 2017
4384c84
Fix merging.
viirya Dec 22, 2017
d864aff
Merge remote-tracking branch 'upstream/master' into SPARK-22856
viirya Feb 21, 2018
8715d32
Add test.
viirya Feb 21, 2018
f59bb19
Use TrueLiteral/FalseLiteral. Add java type and access property to Ex…
viirya Feb 28, 2018
37ae9b0
Address comment.
viirya Mar 1, 2018
0841c4a
Address comment.
viirya Mar 1, 2018
e530f01
Remove ExprType.
viirya Mar 1, 2018
c8c70a9
Merge remote-tracking branch 'upstream/master' into SPARK-22856
viirya Mar 5, 2018
ac2e595
Merge remote-tracking branch 'upstream/master' into SPARK-22856
viirya Apr 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix equality check with string.
  • Loading branch information
@viirya
viirya committed Dec 21, 2017
commit 81c9b6e73ee64adcd8fc931d51f3faa98b892e0b
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ abstract class Expression extends TreeNode[Expression] {
private def reduceCodeSize(ctx: CodegenContext, eval: ExprCode): Unit = {
// TODO: support whole stage codegen too
if (eval.code.trim.length > 1024 && ctx.INPUT_ROW != null && ctx.currentVars == null) {
val setIsNull = if ("false" != eval.isNull && "true" != eval.isNull) {
val setIsNull = if ("false" != s"${eval.isNull}" && "true" != s"${eval.isNull}") {
Copy link
Contributor

@cloud-fan cloud-fan Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be simplified to !eval.isNull.instanceOf[LiteralValue]

Copy link
Member Author

@viirya viirya Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, yea.

val globalIsNull = ctx.addMutableState(ctx.JAVA_BOOLEAN, "globalIsNull")
val localIsNull = eval.isNull
eval.isNull = GlobalValue(globalIsNull)
Expand Down
6 changes: 3 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ case class IsNull(child: Expression) extends UnaryExpression with Predicate {

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
val value = if ("true" == eval.isNull || "false" == eval.isNull) {
val value = if ("true" == s"${eval.isNull}" || "false" == s"${eval.isNull}") {
Copy link
Member

@kiszk kiszk Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if ("true" == eval.isNull || "false" == eval.isNull) {?

Copy link
Contributor

@mgaido91 mgaido91 Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or eval.isNull == Literal("true")? Or even better we can create a LiteralTrue = Literal("true") and equivalent for false and use them?

Copy link
Member Author

@viirya viirya Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do eval.isNull.instanceOf[LiteralValue] as suggested by @cloud-fan below.

LiteralValue(eval.isNull)
} else {
VariableValue(eval.isNull)
Expand Down Expand Up @@ -353,9 +353,9 @@ case class IsNotNull(child: Expression) extends UnaryExpression with Predicate {

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val eval = child.genCode(ctx)
val value = if ("true" == eval.isNull) {
val value = if ("true" == s"${eval.isNull}") {
Copy link
Member

@kiszk kiszk Dec 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

LiteralValue("false")
} else if ("false" == eval.isNull) {
} else if ("false" == s"${eval.isNull}") {
LiteralValue("true")
} else {
StatementValue(s"(!(${eval.isNull}))")
Expand Down