Skip to content

[SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in UnsafeRow. #7682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rxin wants to merge 6 commits into from
Closed

[SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in UnsafeRow. #7682

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
11f80a3
[SPARK-9368][SQL] Support get(ordinal, dataType) generic getter in Un…
rxin Jul 27, 2015
9989064
JoinedRow.
rxin Jul 27, 2015
24a3e46
Added support for DateType/TimestampType.
rxin Jul 27, 2015
fb6ca30
Support BinaryType.
rxin Jul 27, 2015
0f57c55
Reset the changes in ExpressionEvalHelper.
rxin Jul 27, 2015
3063788
Reset the change for real this time.
rxin Jul 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added support for DateType/TimestampType. 
Updated ExpressionEvalHelper to avoid conversion.
  • Loading branch information
@rxin
rxin committed Jul 27, 2015
commit 24a3e4604acb88c1fba83b577de547662285403a
4 changes: 4 additions & 0 deletions sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,10 @@ public Object get(int ordinal, DataType dataType) {
return getDouble(ordinal);
} else if (dataType instanceof DecimalType) {
return getDecimal(ordinal);
} else if (dataType instanceof DateType) {
return getInt(ordinal);
} else if (dataType instanceof TimestampType) {
return getLong(ordinal);
} else if (dataType instanceof StringType) {
return getUTF8String(ordinal);
} else if (dataType instanceof StructType) {
Expand Down
24 changes: 13 additions & 11 deletions ...alyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,17 +160,20 @@ trait ExpressionEvalHelper {
expected: Any,
inputRow: InternalRow = EmptyRow): Unit = {

val plan = generateProject(
val project = generateProject(
GenerateUnsafeProjection.generate(Alias(expression, s"Optimized($expression)")() :: Nil),
expression)

val unsafeRow = plan(inputRow)
// UnsafeRow cannot be compared with GenericInternalRow directly
val actual = FromUnsafeProjection(expression.dataType :: Nil)(unsafeRow)
val expectedRow = InternalRow(expected)
if (actual != expectedRow) {
val input = if (inputRow == EmptyRow) "" else s", input: $inputRow"
fail(s"Incorrect Evaluation: $expression, actual: $actual, expected: $expectedRow$input")
val out = project(inputRow)
val input = if (inputRow == EmptyRow) "" else s", input: $inputRow"

if (expected == null) {
if (!out.isNullAt(0)) {
val actual = out.get(0, expression.dataType)
fail(s"Incorrect Evaluation: $expression, actual: $actual, expected: $expected$input")
}
} else if (out.get(0, expression.dataType) != expected) {
val actual = out.get(0, expression.dataType)
fail(s"Incorrect Evaluation: $expression, actual: $actual, expected: $expected$input")
}
}

Expand Down Expand Up @@ -200,8 +203,7 @@ trait ExpressionEvalHelper {
plan = generateProject(
GenerateUnsafeProjection.generate(Alias(expression, s"Optimized($expression)")() :: Nil),
expression)
actual = FromUnsafeProjection(expression.dataType :: Nil)(
plan(inputRow)).get(0, expression.dataType)
actual = plan(inputRow).get(0, expression.dataType)
assert(checkResult(actual, expected))
}
}