Skip to content

[SPARK-16215][SQL] Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset #13911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 10 commits into from
Closed

[SPARK-16215][SQL] Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset #13911

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1c57972
pass information on containsNull in ArrayType
kiszk Jun 26, 2016
3d78991
generate helper call instead of for-loop
kiszk Jun 26, 2016
90d716f
add test suite
kiszk Jun 26, 2016
0b50242
fix test failure
kiszk Jun 26, 2016
4b132de
update
kiszk Jul 2, 2016
249a1a9
add test suites
kiszk Jul 2, 2016
96b0853
fix potential bug since writePrimitiveTypeArray() may receive Generic…
kiszk Jul 14, 2016
88aad46
rebase
kiszk Nov 3, 2016
25ddd8e
fix test failures
kiszk Nov 3, 2016
6204298
fix test failures
kiszk Nov 4, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add test suite
  • Loading branch information
@kiszk
kiszk committed Nov 3, 2016
commit 90d716f6dafe431ef5cc98592dddbdf0f38defcf
19 changes: 19 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameComplexTypeSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,25 @@ import org.apache.spark.sql.test.SharedSQLContext
class DataFrameComplexTypeSuite extends QueryTest with SharedSQLContext {
import testImplicits._

test("primitive type on array") {
val rows = sparkContext.parallelize(Seq(1, 2), 1).toDF("v").
selectExpr("Array(v + 2, v + 3)")
checkAnswer(rows, Seq(Row(Array(3, 4)), Row(Array(4, 5))))
}

test("primitive type and null on array") {
val rows = sparkContext.parallelize(Seq(1, 2), 1).toDF("v").
selectExpr("Array(v + 2, null, v + 3)")
checkAnswer(rows, Seq(Row(Array(3, null, 4)), Row(Array(4, null, 5))))
}

test("array with null on array") {
val rows = sparkContext.parallelize(Seq(1, 2), 1).toDF("v").
selectExpr("Array(Array(v, v + 1)," +
"null," +
"Array(v, v - 1))").collect
}

test("UDF on struct") {
val f = udf((a: String) => a)
val df = sparkContext.parallelize(Seq((1, 1))).toDF("a", "b")
Expand Down