Skip to content

[SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning #14327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 8 commits into from
Closed

[SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning #14327

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9521a5a
Project shouldn't be pushed down through Sample if it has new output.
viirya Jul 23, 2016
6d1616d
Address comment.
viirya Jul 23, 2016
31a6f6f
Address comments.
viirya Jul 25, 2016
5c4e7ff
Check Dataset result than just calling collect.
viirya Jul 25, 2016
20e9436
Address comments.
viirya Jul 25, 2016
dc70f1d
Use proper name.
viirya Jul 25, 2016
2186b7e
Address comment.
viirya Jul 25, 2016
3e134f1
Fix style.
viirya Jul 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Check Dataset result than just calling collect.
  • Loading branch information
@viirya
viirya committed Jul 25, 2016
commit 5c4e7ffff5bd438ad7abbb37dc84cff9e5036567
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
// After sampling, sampleDF doesn't contain swid=1.
assert(!sampleDF.select("swid").collect.contains(1))
// udfOne should not encounter swid=1.
sampleDF.select(udfOne($"swid")).collect
checkAnswer(sampleDF.select(udfOne($"swid")), List.fill(sampleDF.count.toInt)(Row(1)))
}

test("SPARK-11436: we should rebind right encoder when join 2 datasets") {
Expand Down