Skip to content

[SPARK-17549][sql] Only collect table size stat in driver for cached relation. #15112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vanzin wants to merge 3 commits into from
Closed

[SPARK-17549][sql] Only collect table size stat in driver for cached relation. #15112

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ede3548
[SPARK-17549][sql] Only collect table size stat in driver for cached …
Sep 15, 2016
d356ae3
Add unit test for table size stat.
Sep 15, 2016
dc86fe3
Catch NonFatal() only.
Sep 16, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add unit test for table size stat.
  • Loading branch information
Marcelo Vanzin committed Sep 15, 2016
commit d356ae3d9e6da31784bf9a66dd53c0f5cc2c7e8a
14 changes: 14 additions & 0 deletions ...e/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,4 +232,18 @@ class InMemoryColumnarQuerySuite extends QueryTest with SharedSQLContext {
val columnTypes2 = List.fill(length2)(IntegerType)
val columnarIterator2 = GenerateColumnAccessor.generate(columnTypes2)
}

test("SPARK-17549: cached table size should be correctly calculated") {
val data = spark.sparkContext.parallelize(1 to 10, 5).toDF()
val plan = spark.sessionState.executePlan(data.logicalPlan).sparkPlan
val cached = InMemoryRelation(true, 5, MEMORY_ONLY, plan, None)

// Materialize the data.
val expectedAnswer = data.collect()
checkAnswer(cached, expectedAnswer)

// Check that the right size was calculated.
assert(cached.batchStats.value === expectedAnswer.size * INT.defaultSize)
}
Copy link
Contributor

@yhuai yhuai Sep 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!


}