Skip to content

[SPARK-9390][SQL] create a wrapper for array type #7724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 10 commits into from
Closed

[SPARK-9390][SQL] create a wrapper for array type #7724

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7e283e2
create a wrapper for array type
cloud-fan Jul 28, 2015
d4a38da
remove sizeInBytes and add license
cloud-fan Jul 28, 2015
4346290
address comment
cloud-fan Jul 29, 2015
e719afc
fix hive
cloud-fan Jul 29, 2015
43e9816
fix mllib
cloud-fan Jul 29, 2015
8cb8842
remove element type parameter from getArray
cloud-fan Jul 29, 2015
6dbfa6f
fix hive again...
cloud-fan Jul 29, 2015
f39256c
fix hive...
cloud-fan Jul 29, 2015
661e608
rebase
cloud-fan Jul 30, 2015
d0408a1
fix python
cloud-fan Jul 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix hive again...
  • Loading branch information
@cloud-fan
cloud-fan committed Jul 30, 2015
commit 6dbfa6fc44ac7833b78838db4d7e08ecd3a0a3d7
8 changes: 4 additions & 4 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,8 +591,8 @@ private[hive] object HiveContext {
struct.toSeq.zip(fields).map {
case (v, t) => s""""${t.name}":${toHiveStructString(v, t.dataType)}"""
}.mkString("{", ",", "}")
case (array: ArrayData, ArrayType(typ, _)) =>
array.toArray().map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
case (seq: Seq[_], ArrayType(typ, _)) =>
seq.map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
case (map: Map[_, _], MapType(kType, vType, _)) =>
map.map {
case (key, value) =>
Expand All @@ -614,8 +614,8 @@ private[hive] object HiveContext {
struct.toSeq.zip(fields).map {
case (v, t) => s""""${t.name}":${toHiveStructString(v, t.dataType)}"""
}.mkString("{", ",", "}")
case (array: ArrayData, ArrayType(typ, _)) =>
array.toArray().map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
case (seq: Seq[_], ArrayType(typ, _)) =>
seq.map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
case (map: Map[_, _], MapType(kType, vType, _)) =>
map.map {
case (key, value) =>
Expand Down
23 changes: 17 additions & 6 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,8 @@ import scala.collection.JavaConversions._
* java.sql.Timestamp
* Complex Types =>
* Map: scala.collection.immutable.Map
* List: scala.collection.immutable.Seq
* Struct:
* [[org.apache.spark.sql.catalyst.InternalRow]]
* List: [[org.apache.spark.sql.types.ArrayData]]
* Struct: [[org.apache.spark.sql.catalyst.InternalRow]]
* Union: NOT SUPPORTED YET
* The Complex types plays as a container, which can hold arbitrary data types.
*
Expand Down Expand Up @@ -297,7 +296,10 @@ private[hive] trait HiveInspectors {
}.toMap
case li: StandardConstantListObjectInspector =>
// take the value from the list inspector object, rather than the input data
li.getWritableConstantValue.map(unwrap(_, li.getListElementObjectInspector)).toSeq
val values = li.getWritableConstantValue
.map(unwrap(_, li.getListElementObjectInspector))
.toArray
new GenericArrayData(values)
// if the value is null, we don't care about the object inspector type
case _ if data == null => null
case poi: VoidObjectInspector => null // always be null for void object inspector
Expand Down Expand Up @@ -339,7 +341,10 @@ private[hive] trait HiveInspectors {
}
case li: ListObjectInspector =>
Option(li.getList(data))
.map(_.map(unwrap(_, li.getListElementObjectInspector)).toSeq)
.map { l =>
val values = l.map(unwrap(_, li.getListElementObjectInspector)).toArray
new GenericArrayData(values)
}
.orNull
case mi: MapObjectInspector =>
Option(mi.getMap(data)).map(
Expand Down Expand Up @@ -391,7 +396,13 @@ private[hive] trait HiveInspectors {

case loi: ListObjectInspector =>
val wrapper = wrapperFor(loi.getListElementObjectInspector)
(o: Any) => if (o != null) seqAsJavaList(o.asInstanceOf[Seq[_]].map(wrapper)) else null
(o: Any) => {
if (o != null) {
seqAsJavaList(o.asInstanceOf[ArrayData].toArray().map(wrapper))
} else {
null
}
}

case moi: MapObjectInspector =>
// The Predef.Map is scala.collection.immutable.Map.
Expand Down