Skip to content

[SPARK-23921][SQL] Add array_sort function #21021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 14 commits into from
Closed

[SPARK-23921][SQL] Add array_sort function #21021

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
72f31b3
initial commit
kiszk Apr 10, 2018
a9b6e3b
address review comments
kiszk Apr 10, 2018
d57c14a
adding an update in functions.py
kiszk Apr 10, 2018
9977f64
address review comment
kiszk Apr 10, 2018
172b2c5
address review comments
kiszk Apr 20, 2018
f2798f9
address review comment
kiszk Apr 26, 2018
175d981
address review comments
kiszk Apr 27, 2018
d1b0483
fix test failures in DataFrameFunctionSuite and others
kiszk Apr 29, 2018
04a3ae5
address review comments
kiszk May 1, 2018
9f63a76
address review comments
kiszk May 2, 2018
e3fcaaa
use ctx.addReferenceObj() for DataType in generated code
kiszk May 2, 2018
2ad6bb8
address review comments
kiszk May 4, 2018
2c4404c
Remove redundant cast
kiszk May 4, 2018
21521d8
avoid boxing as possible
kiszk May 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address review comment
  • Loading branch information
@kiszk
kiszk committed Apr 26, 2018
commit f2798f9a6c2a804d6801fbc8b7dbef2755ae5359
18 changes: 9 additions & 9 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,7 @@ case class MapValues(child: Expression)
trait ArraySortUtil extends ExpectsInputTypes with CodegenFallback {
protected def arrayExpression: Expression

// If -1, place null element at the end of the array
// If 1, place null element at the beginning of the array
protected def nullOrder: Int
protected def nullOrder: NullOrder

@transient
private lazy val lt: Comparator[Any] = {
Expand All @@ -140,9 +138,9 @@ trait ArraySortUtil extends ExpectsInputTypes with CodegenFallback {
if (o1 == null && o2 == null) {
0
} else if (o1 == null) {
-1 * nullOrder
nullOrder
} else if (o2 == null) {
1 * nullOrder
-nullOrder
} else {
ordering.compare(o1, o2)
}
Expand All @@ -163,9 +161,9 @@ trait ArraySortUtil extends ExpectsInputTypes with CodegenFallback {
if (o1 == null && o2 == null) {
0
} else if (o1 == null) {
1 * nullOrder
-nullOrder
} else if (o2 == null) {
-1 * nullOrder
nullOrder
} else {
-ordering.compare(o1, o2)
}
Expand All @@ -185,6 +183,8 @@ trait ArraySortUtil extends ExpectsInputTypes with CodegenFallback {

object ArraySortUtil {
type NullOrder = Int
// Least: place null element at the end of the array
// Greatest: place null element at the beginning of the array
Copy link
Member

@ueshin ueshin Apr 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add for ascending order or something?

Copy link
Member Author

@kiszk kiszk Apr 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I will also fix statements.

object NullOrder {
val Least: NullOrder = -1
val Greatest: NullOrder = 1
Expand Down Expand Up @@ -220,7 +220,7 @@ case class SortArray(base: Expression, ascendingOrder: Expression)
override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType, BooleanType)

override def arrayExpression: Expression = base
override def nullOrder: Int = NullOrder.Greatest
override def nullOrder: NullOrder = NullOrder.Least

override def checkInputDataTypes(): TypeCheckResult = base.dataType match {
case ArrayType(dt, _) if RowOrdering.isOrderable(dt) =>
Expand Down Expand Up @@ -356,7 +356,7 @@ case class ArraySort(child: Expression) extends UnaryExpression with ArraySortUt
override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType)

override def arrayExpression: Expression = child
override def nullOrder: Int = NullOrder.Least
override def nullOrder: NullOrder = NullOrder.Greatest

override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case ArrayType(dt, _) if RowOrdering.isOrderable(dt) =>
Expand Down