Skip to content

[SPARK-20254][SQL] Remove unnecessary data conversion for Dataset with primitive array #17568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 19 commits into from
Closed

[SPARK-20254][SQL] Remove unnecessary data conversion for Dataset with primitive array #17568

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1ece803
initial commit
kiszk Apr 7, 2017
c042ff2
addressed review comment
kiszk Apr 8, 2017
264a18b
add an optimizer to remove MapObjects
kiszk Apr 10, 2017
82a1f2b
add test suites to check deserializer in DeserializeToObjectExec
kiszk Apr 10, 2017
cc3c7bd
eliminate no-op AssertNotNull
kiszk Apr 14, 2017
7d43d27
remove unnecessary Mapobjects by checking MapObjects and lambdaVariable
kiszk Apr 14, 2017
3bcee3f
fix scala style error
kiszk Apr 14, 2017
1b24be0
use simpler matchings
kiszk Apr 14, 2017
5174b5e
use non-e2e test suite
kiszk Apr 14, 2017
e608c45
rebase with master
kiszk Apr 15, 2017
2ee4030
fix test failure
kiszk Apr 15, 2017
1d6ab36
address review comments
kiszk Apr 16, 2017
0fd8c25
rebase with master
kiszk Apr 16, 2017
791aad9
address review comments
kiszk Apr 16, 2017
f695e50
address review comments
kiszk Apr 17, 2017
ce6927d
remove MapObject with LambdaVariable if it is for primitive type (non…
kiszk Apr 18, 2017
a6bedee
remove unused import
kiszk Apr 18, 2017
c0dca2b
address review comment
kiszk Apr 18, 2017
8de6915
address review comment
kiszk Apr 18, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use simpler matchings
  • Loading branch information
@kiszk
kiszk committed Apr 16, 2017
commit 1b24be0ca52234997deb549a73cf227339d1ad37
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ case class NullPropagation(conf: SQLConf) extends Rule[LogicalPlan] {
case EqualNullSafe(Literal(null, _), r) => IsNull(r)
case EqualNullSafe(l, Literal(null, _)) => IsNull(l)

case a @ AssertNotNull(c, _) if !c.nullable => c
case _ @ AssertNotNull(c, _) if !c.nullable => c

// For Coalesce, remove null literals.
case e @ Coalesce(children) =>
Expand Down
16 changes: 3 additions & 13 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,21 +108,11 @@ object CombineTypedFilters extends Rule[LogicalPlan] {
* representation of data item. For example back to back map operations.
Copy link
Member

@viirya viirya Apr 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment broken?

*/
object EliminateMapObjects extends Rule[LogicalPlan] {
private def convertDataTypeToArrayClass(dt: DataType): Class[_] = dt match {
case IntegerType => classOf[Array[Int]]
case LongType => classOf[Array[Long]]
case DoubleType => classOf[Array[Double]]
case FloatType => classOf[Array[Float]]
case ShortType => classOf[Array[Short]]
case ByteType => classOf[Array[Byte]]
case BooleanType => classOf[Array[Boolean]]
}

def apply(plan: LogicalPlan): LogicalPlan = plan transform {
Copy link
Contributor

@cloud-fan cloud-fan Apr 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we call plan.transformAllExpressions?

Copy link
Member Author

@kiszk kiszk Apr 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it works. done

case _ @ DeserializeToObject(_ @ Invoke(
MapObjects(_, _, _, LambdaVariable(_, _, _, _), inputData, customCollectionCls, _),
funcName, returnType @ ObjectType(returnCls), arguments, propagateNull, returnNullable),
outputObjAttr, child) if customCollectionCls.isEmpty =>
MapObjects(_, _, _, _ : LambdaVariable, inputData, None, _),
funcName, returnType @ ObjectType(_), arguments, propagateNull, returnNullable),
outputObjAttr, child) =>
DeserializeToObject(Invoke(
inputData, funcName, returnType, arguments, propagateNull, returnNullable),
outputObjAttr, child)
Expand Down