-
Notifications
You must be signed in to change notification settings - Fork 29k
[SQL] Miscellaneous SQL/DF expression changes. #6754
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -91,6 +91,7 @@ trait HiveTypeCoercion { | |
| StringToIntegralCasts :: | ||
| FunctionArgumentConversion :: | ||
| CaseWhenCoercion :: | ||
| IfCoercion :: | ||
| Division :: | ||
| PropagateTypes :: | ||
| ExpectedInputConversion :: | ||
|
|
@@ -652,6 +653,27 @@ trait HiveTypeCoercion { | |
| } | ||
| } | ||
|
|
||
| /** | ||
| * Coerces the type of different branches of If statement to a common type. | ||
| */ | ||
| object IfCoercion extends Rule[LogicalPlan] { | ||
| def apply(plan: LogicalPlan): LogicalPlan = plan transformAllExpressions { | ||
| // Find tightest common type for If, if the true value and false value have different types. | ||
| case i @ If(pred, left, right) if left.dataType != right.dataType => | ||
| findTightestCommonTypeOfTwo(left.dataType, right.dataType).map { widestType => | ||
| val newLeft = if (left.dataType == widestType) left else Cast(left, widestType) | ||
| val newRight = if (right.dataType == widestType) right else Cast(right, widestType) | ||
| i.makeCopy(Array(pred, newLeft, newRight)) | ||
| }.getOrElse(i) // If there is no applicable conversion, leave expression unchanged. | ||
|
|
||
| // Convert If(null literal, _, _) into boolean type. | ||
| // In the optimizer, we should short-circuit this directly into false value. | ||
| case i @ If(pred, left, right) if pred.dataType == NullType => | ||
| println("fireing this rule") | ||
| i.makeCopy(Array(Literal.create(null, BooleanType), left, right)) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why this special case? Also, I'd just use
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "if(null, true, false)" gets a nulltype.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about case i @ If (pred, _, _) if pred.dataType == NullType =>
i.copy(predicate = Literal.create(null, BooleanType))
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 |
||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Casts types according to the expected input types for Expressions that have the trait | ||
| * `ExpectsInputTypes`. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hive support promote types to String for
If, but ourfindTightestCommonTypeOfTwodoesn't support it. There are some other expressions likeCoalesce,CaseWhenneed to promote types to String. #6551 tries to fix it but it's not complete. I'm wondering if hive has a specific rule about string type promotion that we can follow, or is it all about hive's implicit conversions?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we can define new TypeConversion class for Coalesce, If, CaseWhen, and so on. because in Hive these udf function use same one type Conversion rule as https://github.com/apache/hive/blob/ac755ebe26361a4647d53db2a28500f71697b276/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java#L79.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yup that's a good idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a rule to promote to string just for If for this patch. We can generalize it later.