Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use IndexedSeq instead of Array on foldableFieldNames to get better p…
…erformance
  • Loading branch information
jmchung committed Aug 15, 2017
commit 5c69df524ea4a037095005ad9915cac4ca4aedc5
Original file line number Diff line number Diff line change
Expand Up @@ -359,15 +359,15 @@ case class JsonTuple(children: Seq[Expression])
@transient private lazy val jsonExpr: Expression = children.head

// the fields to query are the remaining children
@transient private lazy val fieldExpressions: Array[Expression] = children.tail.toArray
@transient private lazy val fieldExpressions: Seq[Expression] = children.tail

// eagerly evaluate any foldable the field names
@transient private lazy val foldableFieldNames: Array[Option[String]] = {
@transient private lazy val foldableFieldNames: IndexedSeq[Option[String]] = {
fieldExpressions.map {
case expr if expr.foldable => Option(expr.eval()).map(_.asInstanceOf[UTF8String].toString)
case _ => null
}
}
}.toIndexedSeq
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move toIndexedSeq to inner block, i.e. after the map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@viirya Done: (1) remove redundant comment (2) move toIndexedSeq after the map


// and count the number of foldable fields, we'll use this later to optimize evaluation
@transient private lazy val constantFields: Int = foldableFieldNames.count(_ != null)
Expand Down Expand Up @@ -430,6 +430,7 @@ case class JsonTuple(children: Seq[Expression])
}
}

// Array[String]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this line?

val row = Array.ofDim[Any](fieldNames.length)

// start reading through the token stream, looking for any requested field names
Expand Down