[SPARK-8104][SQL] auto alias expressions in analyzer #6647

cloud-fan · 2015-06-04T16:33:17Z

Currently we auto alias expression in parser. However, during parser phase we don't have enough information to do the right alias. For example, Generator that has more than 1 kind of element need MultiAlias, ExtractValue don't need Alias if it's in middle of a ExtractValue chain.

cloud-fan · 2015-06-04T16:34:39Z

The tests will fail as we need to fix the complex field order by issue first. See #5659

cloud-fan · 2015-06-04T16:34:58Z

cc @marmbrus @rxin @chenghao-intel

SparkQA · 2015-06-04T16:41:23Z

Test build #34193 has finished for PR 6647 at commit d471a90.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression with trees.UnaryNode[Expression]
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-05T02:05:57Z

Test build #34241 has finished for PR 6647 at commit 13a92da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

chenghao-intel · 2015-06-05T02:42:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

exprs.map should be enough? We should just resolve the Alias for the root of a given expression tree, right?

chenghao-intel · 2015-06-05T03:14:07Z

For DataFrame API, probably we'd better still keep the existed naming pattern, as we don't want to confusing the users when upgrade the spark.

SparkQA · 2015-06-05T10:08:06Z

Test build #34268 has finished for PR 6647 at commit 9f8578e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct(val scalingVec: Vector) extends VectorTransformer
- trait TypeCheckResult
- case class TypeCheckFailure(message: String) extends TypeCheckResult
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue
- abstract class UnaryArithmetic extends UnaryExpression
- case class UnaryMinus(child: Expression) extends UnaryArithmetic
- case class Sqrt(child: Expression) extends UnaryArithmetic
- case class Abs(child: Expression) extends UnaryArithmetic
- case class BitwiseNot(child: Expression) extends UnaryArithmetic
- case class MaxOf(left: Expression, right: Expression) extends BinaryArithmetic
- case class MinOf(left: Expression, right: Expression) extends BinaryArithmetic
- case class Atan2(left: Expression, right: Expression)
- case class Hypot(left: Expression, right: Expression)
- case class EqualTo(left: Expression, right: Expression) extends BinaryComparison

SparkQA · 2015-06-06T03:46:36Z

Test build #34353 has finished for PR 6647 at commit 5dcf292.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-18T15:42:49Z

Test build #35138 has finished for PR 6647 at commit 02ed8a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-19T05:44:54Z

Test build #35226 has finished for PR 6647 at commit 6ac6c0e.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

cloud-fan · 2015-06-21T09:08:00Z

retest this please.

SparkQA · 2015-06-21T10:23:22Z

Test build #35386 has finished for PR 6647 at commit 6ac6c0e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-21T14:46:59Z

Test build #35394 has finished for PR 6647 at commit 3acc43a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-21T17:34:43Z

Test build #35399 has finished for PR 6647 at commit 73a90cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-21T19:27:27Z

Test build #35407 has finished for PR 6647 at commit 3a66e4b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-21T21:02:54Z

Test build #35413 has finished for PR 6647 at commit 5b5786d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T02:05:08Z

Test build #35424 has finished for PR 6647 at commit e2fb9e1.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T05:42:56Z

Test build #35432 has finished for PR 6647 at commit be5ee2d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T08:08:14Z

Test build #35436 has finished for PR 6647 at commit d464641.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T10:12:43Z

Test build #35438 has finished for PR 6647 at commit a0a5401.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T13:22:41Z

Test build #35443 has finished for PR 6647 at commit b048f96.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

SparkQA · 2015-06-22T15:29:46Z

Test build #35449 has finished for PR 6647 at commit 552eba4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedAlias(child: Expression) extends NamedExpression
- abstract class ExtractValueWithStruct extends ExtractValue

marmbrus · 2015-06-22T19:11:48Z

Thanks for cleaning this up. Merging to master!

Currently we auto alias expression in parser. However, during parser phase we don't have enough information to do the right alias. For example, Generator that has more than 1 kind of element need MultiAlias, ExtractValue don't need Alias if it's in middle of a ExtractValue chain. Author: Wenchen Fan <[email protected]> Closes apache#6647 from cloud-fan/alias and squashes the following commits: 552eba4 [Wenchen Fan] fix python 5b5786d [Wenchen Fan] fix agg 73a90cb [Wenchen Fan] fix case-preserve of ExtractValue 4cfd23c [Wenchen Fan] fix order by d18f401 [Wenchen Fan] refine 9f07359 [Wenchen Fan] address comments 39c1aef [Wenchen Fan] small fix 33640ec [Wenchen Fan] auto alias expressions in analyzer

chenghao-intel reviewed Jun 5, 2015
View reviewed changes

cloud-fan force-pushed the alias branch from 5dcf292 to 02ed8a3 Compare June 18, 2015 15:18

cloud-fan force-pushed the alias branch from 02ed8a3 to 6ac6c0e Compare June 19, 2015 05:28

cloud-fan added 6 commits June 22, 2015 00:10

auto alias expressions in analyzer

33640ec

small fix

39c1aef

address comments

9f07359

refine

d18f401

fix order by

4cfd23c

fix case-preserve of ExtractValue

73a90cb

cloud-fan force-pushed the alias branch from 3acc43a to 73a90cb Compare June 21, 2015 16:20

fix agg

5b5786d

cloud-fan force-pushed the alias branch from 3a66e4b to 5b5786d Compare June 21, 2015 19:45

cloud-fan force-pushed the alias branch from e2fb9e1 to be5ee2d Compare June 22, 2015 05:17

cloud-fan force-pushed the alias branch from be5ee2d to d464641 Compare June 22, 2015 06:51

cloud-fan force-pushed the alias branch from d464641 to a0a5401 Compare June 22, 2015 09:39

cloud-fan force-pushed the alias branch from a0a5401 to b048f96 Compare June 22, 2015 11:30

fix python

552eba4

cloud-fan force-pushed the alias branch from b048f96 to 552eba4 Compare June 22, 2015 13:35

asfgit closed this in da7bbb9 Jun 22, 2015

[SPARK-8104][SQL] auto alias expressions in analyzer #6647

[SPARK-8104][SQL] auto alias expressions in analyzer #6647

Uh oh!

Conversation

cloud-fan commented Jun 4, 2015

Uh oh!

cloud-fan commented Jun 4, 2015

Uh oh!

cloud-fan commented Jun 4, 2015

Uh oh!

SparkQA commented Jun 4, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

chenghao-intel Jun 5, 2015

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 5, 2015

Uh oh!

SparkQA commented Jun 6, 2015

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

cloud-fan commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 21, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

marmbrus commented Jun 22, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants