Skip to content

[SPARK-20311][SQL] Support aliases for table value functions #17928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
maropu wants to merge 4 commits into from
Closed

[SPARK-20311][SQL] Support aliases for table value functions #17928

maropu wants to merge 4 commits into from

Conversation

@maropu
Copy link
Member

@maropu maropu commented May 10, 2017

What changes were proposed in this pull request?

This pr added parsing rules to support aliases in table value functions.
The previous pr (#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in SQLQueryTestSuite.

How was this patch tested?

Added tests in PlanParserSuite and SQLQueryTestSuite.

cloud-fan
cloud-fan reviewed May 10, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out Outdated
Copy link
Contributor

@cloud-fan cloud-fan May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we breaking existing queries?

Copy link
Member Author

@maropu maropu May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, my bad. I'll check again.

Copy link
Member Author

@maropu maropu May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76712 has finished for PR 17928 at commit 47b923b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76716 has finished for PR 17928 at commit 0904fc9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@maropu
Copy link
Member Author

maropu commented May 10, 2017

@cloud-fan ok, could you check again? Thanks!

@cloud-fan
Copy link
Contributor

cloud-fan commented May 10, 2017

LGTM, cc @gatorsmile to take another look

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
;

tableAlias
: (AS? strictIdentifier identifierList?)?
Copy link
Member

@gatorsmile gatorsmile May 10, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also hits another bug in inline tables. Maybe you also can include the following query in the test case inline-table.sql?

sql("SELECT * FROM VALUES (\"one\", 1), (\"three\", null) CROSS JOIN VALUES (\"one\", 1), (\"three\", null)")

Copy link
Member Author

@maropu maropu May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala Outdated

test("SPARK-20311 range(N) as alias") {
assertEqual(
"select * from range(10) AS t",
Copy link
Member

@gatorsmile gatorsmile May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: SELECT * FROM range(10) AS t
BTW, we prefer to use upper case for the SQL keywords.

Copy link
Member

@gatorsmile gatorsmile May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also can update the similar issues in your test cases.

Copy link
Member Author

@maropu maropu May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@gatorsmile
Copy link
Member

gatorsmile commented May 10, 2017

LGTM

@SparkQA
Copy link

SparkQA commented May 10, 2017

Test build #76737 has finished for PR 17928 at commit 29281b1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql Outdated
EXPLAIN select * from RaNgE(2);

-- cross-join table valued functions
SET spark.sql.crossJoin.enabled=true;
Copy link
Member

@gatorsmile gatorsmile May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove this line? If we specify CROSS JOIN in the query, no need to set this parm.

Copy link
Member Author

@maropu maropu May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76770 has finished for PR 17928 at commit 54a05da.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed May 11, 2017
View reviewed changes
sql/core/src/test/resources/sql-tests/inputs/inline-table.sql Outdated
select * from values (timestamp('1991-12-06 00:00:00.0'), array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0'))) as data(a, b);

-- cross-join inline tables
SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN VALUES ('one', 1), ('three', null);
Copy link
Contributor

@cloud-fan cloud-fan May 11, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this expose the bug? If we treat CROSS as an alias, we still get the same result. how about we run EXPLAIN?

Copy link
Member Author

@maropu maropu May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, ok

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76784 has started for PR 17928 at commit a7a732b.

@cloud-fan
Copy link
Contributor

cloud-fan commented May 11, 2017

retest this please

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76791 has finished for PR 17928 at commit a7a732b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@asfgit asfgit closed this in 8c67aa7 May 11, 2017
@gatorsmile
Copy link
Member

gatorsmile commented May 11, 2017

Thanks! Merging to master.

lycplus pushed a commit to lycplus/spark that referenced this pull request May 24, 2017
@maropu @lycplus
[SPARK-20311][SQL] Support aliases for table value functions
10e528a 
## What changes were proposed in this pull request?
This pr added parsing rules to support aliases in table value functions.
The previous pr (apache#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in `SQLQueryTestSuite`.

## How was this patch tested?
Added tests in `PlanParserSuite` and `SQLQueryTestSuite`.

Author: Takeshi Yamamuro <[email protected]>

Closes apache#17928 from maropu/SPARK-20311-3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

@gatorsmile gatorsmile gatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@maropu @SparkQA @cloud-fan @gatorsmile