Skip to content

[PARK-6489][SQL] add column pruning for Generate #8268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cloud-fan wants to merge 2 commits into from
Closed

[PARK-6489][SQL] add column pruning for Generate #8268

cloud-fan wants to merge 2 commits into from

Conversation

@cloud-fan
Copy link
Contributor

@cloud-fan cloud-fan commented Aug 18, 2015

This PR takes over #5358

@SparkQA
Copy link

SparkQA commented Aug 18, 2015

Test build #41101 has finished for PR 8268 at commit 8fdee4a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Aug 18, 2015

cc @marmbrus @rxin

marmbrus
marmbrus reviewed Aug 18, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
Copy link
Contributor

@marmbrus marmbrus Aug 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also handle the join case when there is a project on top of the generate (i.e. push the project through the generate)?

@SparkQA
Copy link

SparkQA commented Aug 19, 2015

Test build #41207 has finished for PR 8268 at commit bf45cac.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Aug 19, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 19, 2015

Test build #41242 has finished for PR 8268 at commit 2ce282e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

yhuai
yhuai reviewed Aug 19, 2015
View reviewed changes
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
Copy link
Contributor

@yhuai yhuai Aug 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can also have a test for val input = LocalRelation('a.int, 'b.array(StringType)). Basically, we test turning join to false and then we add a project as the child of Generate.

@yhuai
Copy link
Contributor

yhuai commented Aug 19, 2015

LGTM.

@marmbrus
Copy link
Contributor

marmbrus commented Aug 19, 2015

Thanks! Merging to master and 1.5

asfgit pushed a commit that referenced this pull request Aug 19, 2015
@cloud-fan @marmbrus
[SPARK-6489] [SQL] add column pruning for Generate
5c749c8 
This PR takes over #5358

Author: Wenchen Fan <[email protected]>

Closes #8268 from cloud-fan/6489.

(cherry picked from commit b0dbaec)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in b0dbaec Aug 19, 2015
@cloud-fan cloud-fan deleted the 6489 branch August 20, 2015 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cloud-fan @SparkQA @yhuai @marmbrus