Skip to content

[SPARK-24119][SQL]Add interpreted execution to SortPrefix expression #21231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bersprockets wants to merge 13 commits into from
Closed

[SPARK-24119][SQL]Add interpreted execution to SortPrefix expression #21231

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
280b941
Checkpoint changes
bersprockets Apr 28, 2018
4a810ae
Checkpoint commit
bersprockets Apr 29, 2018
d1a7e22
Checkpoint commit
bersprockets Apr 29, 2018
4dbb0b7
Comment on testing oddity
bersprockets Apr 29, 2018
227b6ac
Add boolean test for Sortprefix
bersprockets Apr 30, 2018
bde7b5e
Fix local timezone issue with tests
bersprockets May 4, 2018
71007e5
Respond to review comments
bersprockets May 4, 2018
1060a66
Small cleanup
bersprockets May 4, 2018
dc9a070
Fix test
bersprockets May 7, 2018
2044aed
Split decimal handling function into two
bersprockets May 8, 2018
b81419f
Collapse function creation for Integral, Date, and Timestamp types
bersprockets May 8, 2018
28f1b70
Remove extraneous empty line
bersprockets May 8, 2018
590ba26
Move out of closure those variable evaluations that need to happen on…
bersprockets May 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix local timezone issue with tests
  • Loading branch information
@bersprockets
bersprockets committed May 4, 2018
commit bde7b5ea925c8a15e63d2cfadb248c686f0e1e06
6 changes: 5 additions & 1 deletion .../src/test/scala/org/apache/spark/sql/catalyst/expressions/SortOrderExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.sql.catalyst.expressions

import java.sql.{Date, Timestamp}
import java.util.TimeZone

import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.types._
Expand All @@ -27,6 +28,9 @@ import org.apache.spark.util.collection.unsafe.sort.PrefixComparators.{BinaryPre
class SortOrderExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {

test("SortPrefix") {
// Explicitly choose a time zone, since Date objects can create different values depending on
// local time zone of the machine on which the test is running
TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"))
Copy link
Member

@kiszk kiszk May 6, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we need to restore the original time zone after the test?

Copy link
Contributor Author

@bersprockets bersprockets May 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kiszk Maybe not a nit. I should fix that.

val b1 = Literal.create(false, BooleanType)
val b2 = Literal.create(true, BooleanType)
val i1 = Literal.create(20132983, IntegerType)
Expand Down Expand Up @@ -57,7 +61,7 @@ class SortOrderExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(SortPrefix(SortOrder(l2, Ascending)), -20132983L)
// For some reason, the Literal.create code gives us the number of days since the epoch
checkEvaluation(SortPrefix(SortOrder(d1, Ascending)), 17649L)
checkEvaluation(SortPrefix(SortOrder(t1, Ascending)), millis*1000)
checkEvaluation(SortPrefix(SortOrder(t1, Ascending)), millis * 1000)
checkEvaluation(SortPrefix(SortOrder(f1, Ascending)),
DoublePrefixComparator.computePrefix(f1.value.asInstanceOf[Float].toDouble))
checkEvaluation(SortPrefix(SortOrder(f2, Ascending)),
Expand Down