Skip to content

[SPARK-8010][SQL]Promote types to StringType as implicit conversion in non-binary expression of HiveTypeCoercion #6551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
OopsOutOfMemory wants to merge 6 commits into from
Closed

[SPARK-8010][SQL]Promote types to StringType as implicit conversion in non-binary expression of HiveTypeCoercion #6551

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
403809c
promote non-string to string when can not found tighestCommonTypeOfTwo
OopsOutOfMemory Jun 5, 2015
95cbd58
fix style
OopsOutOfMemory Jun 5, 2015
df365d2
refine
OopsOutOfMemory Jun 5, 2015
4cd5618
limit the data type to primitive type
OopsOutOfMemory Jun 5, 2015
6018613
convert function to method
OopsOutOfMemory Jun 8, 2015
7a209d7
rebase master
OopsOutOfMemory Jun 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
rebase master
  • Loading branch information
@OopsOutOfMemory
OopsOutOfMemory committed Jun 12, 2015
commit 7a209d7d02af0fc6427a967ac35145bc5e3e3383
19 changes: 3 additions & 16 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,27 +68,14 @@ object HiveTypeCoercion {
}

/**
* Implicit promote the AtomicType to StringType if
* one of the data type in (dt1, dt2) is StringType, and the other is not either
* BooleanType or BinaryType, the TightestCommonType should be StringType
* eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
* 2. Coalesce(null, dt1, dt2)
* Similar to [[findTightestCommonType]], if can not find the TightestCommonType, try to use
* [[findTightestCommonTypeToString]] to find the TightestCommonType.
*/
private def promoteToStringType(t1: DataType, t2: DataType): Option[DataType] = (t1, t2) match {
case (t1: StringType, t2: AtomicType) if (t2 != BinaryType && t2 != BooleanType) =>
Some(StringType)

case (t1: AtomicType, t2: StringType) if (t1 != BinaryType && t1 != BooleanType) =>
Some(StringType)

case _ => None
}

private def findTightestCommonTypeAndPromoteToString(types: Seq[DataType]): Option[DataType] = {
types.foldLeft[Option[DataType]](Some(NullType))((r, c) => r match {
case None => None
case Some(d) =>
findTightestCommonTypeOfTwo(d, c).orElse(promoteToStringType(d, c))
findTightestCommonTypeOfTwo(d, c).orElse(findTightestCommonTypeToString(d, c))
})
}

Expand Down