[SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint

[yaooqinn]

What changes were proposed in this pull request?

This PR makes the below case work well.

select a b from values(1) t(a) distribute by a;

== Parsed Logical Plan ==
'RepartitionByExpression ['a]
+- 'Project ['a AS b#42]
   +- 'SubqueryAlias t
      +- 'UnresolvedInlineTable [a], [List(1)]

== Analyzed Logical Plan ==
org.apache.spark.sql.AnalysisException: cannot resolve 'a' given input columns: [b]; line 1 pos 62;
'RepartitionByExpression ['a]
+- Project [a#48 AS b#42]
   +- SubqueryAlias t
      +- LocalRelation [a#48]

Why are the changes needed?

bugfix

Does this PR introduce any user-facing change?

yes, the original attributes can be used in distribute by / cluster by and hints like /*+ REPARTITION(3, c) */

How was this patch tested?

new tests

[yaooqinn]

cc @cloud-fan @maropu @HyukjinKwon thanks for reviewing

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42753/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42753/

[cloud-fan]

is this a regression? It seems to me that this should fail, as it's similar to sql("select a b from values(1) t(a)").repartitionBy("a")

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42756/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42756/

[yaooqinn]

is this a regression?

this is not a regression, but hive supports it.

It seems to me that this should fail, as it's similar to sql("select a b from values(1) t(a)").repartitionBy("a")

For resolving attributes, distribute by and cluster by clauses should behave the same as sort by and group by?

[SparkQA]

Test build #138231 has finished for PR 32465 at commit 0c711e3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #138234 has finished for PR 32465 at commit 03ed3a5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @yaooqinn .
As you mentioned, this is not a regression and didn't work before. I believe this should be merged as a new improvement to Apache Spark 3.2.0.

[dongjoon-hyun]

If you don't mind, could you add a test case for HINT explicitly?

[dongjoon-hyun]

Thank you for adding a test, @yaooqinn .

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42795/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42795/

[yaooqinn]

Thank you for adding a test, @yaooqinn .

It's my pleasure

[yaooqinn]

Hi, @yaooqinn .
As you mentioned, this is not a regression and didn't work before. I believe this should be merged as a new improvement to Apache Spark 3.2.0.

OK~

[SparkQA]

Test build #138273 has finished for PR 32465 at commit 2ce0fb6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42810/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42810/

[dongjoon-hyun]

Merged to master. Thank you, @yaooqinn and @cloud-fan .

[SparkQA]

Test build #138288 has finished for PR 32465 at commit 5163262.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.