Skip to content

[SPARK-24734][SQL] Fix type coercions and nullabilities of nested data types of some functions. #21704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ueshin wants to merge 21 commits into from
Closed

[SPARK-24734][SQL] Fix type coercions and nullabilities of nested data types of some functions. #21704

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d87a8c6
Fix containsNull of Concat for array type.
ueshin Jul 3, 2018
30d5aed
Add a test.
ueshin Jul 4, 2018
2e624df
Revert "Fix containsNull of Concat for array type."
ueshin Jul 9, 2018
fa73b32
Merge branch 'master' into issues/SPARK-24734/concat_containsnull
ueshin Jul 9, 2018
da0702b
Suppress unneeded casts.
ueshin Jul 9, 2018
b31e401
Fix `Concat`.
ueshin Jul 9, 2018
7a838b0
Fix `CreateArray`.
ueshin Jul 9, 2018
01c9ff3
Fix `CreateMap`.
ueshin Jul 9, 2018
b2ca587
Fix `Coalesce`.
ueshin Jul 9, 2018
3d8891e
Fix `Least` and `Greatest`.
ueshin Jul 9, 2018
1fa692a
Merge branch 'master' into issues/SPARK-24734/concat_containsnull
ueshin Jul 9, 2018
444383d
Fix `MapConcat`.
ueshin Jul 9, 2018
3e1f7e4
Fix udts.
ueshin Jul 10, 2018
2c54e38
Rename to `findCommonTypeDifferentOnlyInNullFlags`.
ueshin Jul 10, 2018
5f1b865
Merge branch 'master' into issues/SPARK-24734/concat_containsnull
ueshin Jul 11, 2018
2ab025f
Use `ComplexTypeMergingExpression` trait.
ueshin Jul 11, 2018
db254e5
Fix udts.
ueshin Jul 11, 2018
b412f7b
Merge branch 'master' into issues/SPARK-24734/concat_containsnull
ueshin Jul 12, 2018
f701242
Make `ArraySetLike` `ComplexTypeMergingExpression`.
ueshin Jul 12, 2018
5115961
Use `TypeCoercion.haveSameType`.
ueshin Jul 13, 2018
e489e8b
Revert "Make `ArraySetLike` `ComplexTypeMergingExpression`."
ueshin Jul 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix udts.
  • Loading branch information
@ueshin
ueshin committed Jul 11, 2018
commit db254e582775d8715a945ab1b48fb8587c8e934a
6 changes: 0 additions & 6 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,12 +169,6 @@ object TypeCoercion {
}
case _ => None
}
case (udt1: UserDefinedType[_], udt2: UserDefinedType[_]) =>
findTypeFunc(udt1.sqlType, udt2.sqlType)
case (udt1: UserDefinedType[_], _) =>
findTypeFunc(udt1.sqlType, t2)
case (_, udt2: UserDefinedType[_]) =>
findTypeFunc(t1, udt2.sqlType)
case _ => None
}

Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ object Literal {
case map: MapType => create(Map(), map)
case struct: StructType =>
create(InternalRow.fromSeq(struct.fields.map(f => default(f.dataType).value)), struct)
case udt: UserDefinedType[_] => default(udt.sqlType)
case udt: UserDefinedType[_] => Literal(default(udt.sqlType).value, udt)
case other =>
throw new RuntimeException(s"no default for type $dataType")
}
Expand Down