Skip to content

[SPARK-7109] [SQL] Push down left side filter for left semi join #5677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
scwf wants to merge 5 commits into from
Closed

[SPARK-7109] [SQL] Push down left side filter for left semi join #5677

scwf wants to merge 5 commits into from

Conversation

@scwf
Copy link
Contributor

@scwf scwf commented Apr 24, 2015

Now in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.

@scwf
Copy link
Contributor Author

scwf commented Apr 24, 2015

i will try to add a test case for this

@SparkQA
Copy link

SparkQA commented Apr 24, 2015

Test build #30911 has finished for PR 5677 at commit ebadaa9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@scwf
Copy link
Contributor Author

scwf commented Apr 24, 2015

added test

@SparkQA
Copy link

SparkQA commented Apr 24, 2015

Test build #30941 has finished for PR 5677 at commit 8f48a3d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@scwf
Copy link
Contributor Author

scwf commented Apr 25, 2015

Jenkins, retest this please

@SparkQA
Copy link

SparkQA commented Apr 27, 2015

Test build #30980 has started for PR 5677 at commit 8f48a3d.

@scwf
Copy link
Contributor Author

scwf commented Apr 29, 2015

Jenkins, retest this please

@SparkQA
Copy link

SparkQA commented Apr 29, 2015

Test build #31350 has finished for PR 5677 at commit 8f48a3d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@scwf
Copy link
Contributor Author

scwf commented Apr 30, 2015

It's very strange that compile with hive12 passed but failed for hive13, and my locally building on hive13 is ok

@SparkQA
Copy link

SparkQA commented Apr 30, 2015

Test build #31366 has finished for PR 5677 at commit d68a053.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@scwf
Copy link
Contributor Author

scwf commented Apr 30, 2015

oh, my bad, i need rebase my branch since master renamed apply -> execute

@SparkQA
Copy link

SparkQA commented Apr 30, 2015

Test build #31368 has finished for PR 5677 at commit 483d205.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Param[T] (val parent: Params, val name: String, val doc: String, val isValid: T => Boolean)
    • class DoubleParam(parent: Params, name: String, doc: String, isValid: Double => Boolean)
    • class IntParam(parent: Params, name: String, doc: String, isValid: Int => Boolean)
    • class FloatParam(parent: Params, name: String, doc: String, isValid: Float => Boolean)
    • class LongParam(parent: Params, name: String, doc: String, isValid: Long => Boolean)
    • class BooleanParam(parent: Params, name: String, doc: String) // No need for isValid
    • case class ParamPair[T](param: Param[T], value: T)
  • This patch does not change any dependencies.

@chenghao-intel
Copy link
Contributor

chenghao-intel commented Apr 30, 2015

LGTM

@scwf
Copy link
Contributor Author

scwf commented Apr 30, 2015

@rxin @marmbrus any comment here?

@marmbrus
Copy link
Contributor

marmbrus commented May 1, 2015

Thanks, merged to master.

@asfgit asfgit closed this in a0d8a61 May 1, 2015
@scwf scwf deleted the leftsemi branch May 1, 2015 01:20
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@scwf @jeanlyn
[SPARK-7109] [SQL] Push down left side filter for left semi join
d7cbfdf 
Now in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.

Author: wangfei <[email protected]>
Author: scwf <[email protected]>

Closes apache#5677 from scwf/leftsemi and squashes the following commits:

483d205 [wangfei] update with master to fix compile issue
82df0e1 [wangfei] Merge branch 'master' of https://github.com/apache/spark into leftsemi
d68a053 [wangfei] added apply
8f48a3d [scwf] added test
ebadaa9 [wangfei] left filter push down for left semi join
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@scwf @jeanlyn
[SPARK-7109] [SQL] Push down left side filter for left semi join
7e2c6e0 
Now in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.

Author: wangfei <[email protected]>
Author: scwf <[email protected]>

Closes apache#5677 from scwf/leftsemi and squashes the following commits:

483d205 [wangfei] update with master to fix compile issue
82df0e1 [wangfei] Merge branch 'master' of https://github.com/apache/spark into leftsemi
d68a053 [wangfei] added apply
8f48a3d [scwf] added test
ebadaa9 [wangfei] left filter push down for left semi join
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@scwf @nemccarthy
[SPARK-7109] [SQL] Push down left side filter for left semi join
143aa83 
Now in spark sql optimizer we only push down right side filter for left semi join, actually we can push down left side filter because left semi join is doing filter on left table essentially.

Author: wangfei <[email protected]>
Author: scwf <[email protected]>

Closes apache#5677 from scwf/leftsemi and squashes the following commits:

483d205 [wangfei] update with master to fix compile issue
82df0e1 [wangfei] Merge branch 'master' of https://github.com/apache/spark into leftsemi
d68a053 [wangfei] added apply
8f48a3d [scwf] added test
ebadaa9 [wangfei] left filter push down for left semi join
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@scwf @SparkQA @chenghao-intel @marmbrus