Skip to content

[SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning #14327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 8 commits into from
Closed

[SPARK-16686][SQL] Remove PushProjectThroughSample since it is handled by ColumnPruning #14327

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9521a5a
Project shouldn't be pushed down through Sample if it has new output.
viirya Jul 23, 2016
6d1616d
Address comment.
viirya Jul 23, 2016
31a6f6f
Address comments.
viirya Jul 25, 2016
5c4e7ff
Check Dataset result than just calling collect.
viirya Jul 25, 2016
20e9436
Address comments.
viirya Jul 25, 2016
dc70f1d
Use proper name.
viirya Jul 25, 2016
2186b7e
Address comment.
viirya Jul 25, 2016
3e134f1
Fix style.
viirya Jul 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address comment.
  • Loading branch information
@viirya
viirya committed Jul 25, 2016
commit 2186b7ef95a2a2fc3dd047fb256d6c71fdd6c6ad
7 changes: 2 additions & 5 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.reflect.runtime.universe.TypeTag

import org.apache.spark.sql.catalyst.analysis
import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
Expand Down Expand Up @@ -353,8 +352,7 @@ class ColumnPruningSuite extends PlanTest {
val originalQuery =
Copy link
Contributor

@cloud-fan cloud-fan Jul 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the style should be:

val query1 = xxxx.analyze
val optimized1 = ...
val expected1 = ...

val query2 ....
...

Copy link
Member Author

@viirya viirya Jul 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

Sample(0.0, 0.6, false, 11L, x)().select('a)

val originalQueryAnalyzed =
EliminateSubqueryAliases(analysis.SimpleAnalyzer.execute(originalQuery))
val originalQueryAnalyzed = originalQuery.analyze

val optimized = Optimize.execute(originalQueryAnalyzed)

Expand All @@ -366,8 +364,7 @@ class ColumnPruningSuite extends PlanTest {
val originalQuery2 =
Sample(0.0, 0.6, false, 11L, x)().select('a as 'aa)

val originalQueryAnalyzed2 =
EliminateSubqueryAliases(analysis.SimpleAnalyzer.execute(originalQuery2))
val originalQueryAnalyzed2 = originalQuery2.analyze

val optimized2 = Optimize.execute(originalQueryAnalyzed2)

Expand Down