Skip to content

[SPARK-30940][SQL] Remove attributeId in auto-generated arguments when Explain SQL query #27685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Eric5553 wants to merge 4 commits into from
Closed

[SPARK-30940][SQL] Remove attributeId in auto-generated arguments when Explain SQL query #27685

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
19ddd4f
Only remote auto-generated attrId
Eric5553 Mar 2, 2020
62065ac
refine code to provide better atomicity
Eric5553 Mar 3, 2020
ea136d1
fix pattern match null
Eric5553 Mar 3, 2020
c7daaa6
remove one function in Expression
Eric5553 Mar 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refine code to provide better atomicity
  • Loading branch information
@Eric5553
Eric5553 committed Apr 9, 2020
commit 62065acdb00d291fbdebdf7d84d64566b4772809
8 changes: 6 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,18 +263,22 @@ abstract class Expression extends TreeNode[Expression] {

protected def flatArguments: Iterator[Any] = stringArgs.flatMap {
case t: Iterable[_] => t
case e: Expression => e.argumentString :: Nil
case single => single :: Nil
}

protected def flatArgumentStrings: Iterator[String] = flatArguments.map {
case e: Expression => e.argumentString
case arg: Any => arg.toString
}

// Marks this as final, Expression.verboseString should never be called, and thus shouldn't be
// overridden by concrete classes.
final override def verboseString(maxFields: Int): String = simpleString(maxFields)

override def simpleString(maxFields: Int): String = toString

override def toString: String = prettyName + truncatedString(
flatArguments.toSeq, "(", ", ", ")", SQLConf.get.maxToStringFields)
flatArgumentStrings.toSeq, "(", ", ", ")", SQLConf.get.maxToStringFields)

def argumentString: String = toString
Copy link
Contributor

@cloud-fan cloud-fan Mar 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to only add one method? I'm worried about adding to many methods to the framework.

Copy link
Contributor Author

@Eric5553 Eric5553 Mar 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the argumentString is needed because AttributeReference already overwrite toString, thus we need the new abstract string function to switch to non-exprid format. For flatArgumentStrings, it only have two callers. I refactored the toAggString of AggregateFunction, then we don't need to add the method flatArgumentStrings in Expression but just implement it within toString. See commit b74c500.


Expand Down
2 changes: 1 addition & 1 deletion ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ abstract class AggregateFunction extends Expression {
/** String representation used in explain plans. */
def toAggString(isDistinct: Boolean): String = {
val start = if (isDistinct) "(distinct " else "("
prettyName + flatArguments.mkString(start, ", ", ")")
prettyName + flatArgumentStrings.mkString(start, ", ", ")")
}
}

Expand Down
20 changes: 10 additions & 10 deletions sql/core/src/test/resources/sql-tests/results/explain.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Results [2]: [key#x, max#x]

(6) Exchange
Input [2]: [key#x, max#x]
Arguments: hashpartitioning(key#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, 4), true, [id=#x]

(7) HashAggregate [codegen id : 2]
Input [2]: [key#x, max#x]
Expand Down Expand Up @@ -158,7 +158,7 @@ Results [2]: [key#x, max#x]

(6) Exchange
Input [2]: [key#x, max#x]
Arguments: hashpartitioning(key#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, 4), true, [id=#x]

(7) HashAggregate [codegen id : 2]
Input [2]: [key#x, max#x]
Expand Down Expand Up @@ -246,7 +246,7 @@ Results [2]: [key#x, val#x]

(11) Exchange
Input [2]: [key#x, val#x]
Arguments: hashpartitioning(key#x, val#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, val, 4), true, [id=#x]

(12) HashAggregate [codegen id : 4]
Input [2]: [key#x, val#x]
Expand Down Expand Up @@ -832,7 +832,7 @@ Results [2]: [key#x, max#x]

(6) Exchange
Input [2]: [key#x, max#x]
Arguments: hashpartitioning(key#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, 4), true, [id=#x]

(7) HashAggregate [codegen id : 4]
Input [2]: [key#x, max#x]
Expand Down Expand Up @@ -917,7 +917,7 @@ Input [2]: [key#x, val#x]
(3) HashAggregate
Input [2]: [key#x, val#x]
Keys: []
Functions [3]: [partial_count(val#x), partial_sum(cast(key#x as bigint)), partial_count(key#x) FILTER (WHERE (val#x > 1))]
Functions [3]: [partial_count(val), partial_sum(cast(key#x as bigint)), partial_count(key) FILTER (WHERE (val#x > 1))]
Aggregate Attributes [3]: [count#xL, sum#xL, count#xL]
Results [3]: [count#xL, sum#xL, count#xL]

Expand All @@ -928,9 +928,9 @@ Arguments: SinglePartition, true, [id=#x]
(5) HashAggregate [codegen id : 2]
Input [3]: [count#xL, sum#xL, count#xL]
Keys: []
Functions [3]: [count(val#x), sum(cast(key#x as bigint)), count(key#x)]
Aggregate Attributes [3]: [count(val#x)#xL, sum(cast(key#x as bigint))#xL, count(key#x)#xL]
Results [2]: [(count(val#x)#xL + sum(cast(key#x as bigint))#xL) AS TOTAL#xL, count(key#x)#xL AS count(key) FILTER (WHERE (val > 1))#xL]
Functions [3]: [count(val), sum(cast(key#x as bigint)), count(key)]
Aggregate Attributes [3]: [count(val)#xL, sum(cast(key#x as bigint))#xL, count(key)#xL]
Results [2]: [(count(val)#xL + sum(cast(key#x as bigint))#xL) AS TOTAL#xL, count(key)#xL AS count(key) FILTER (WHERE (val > 1))#xL]


-- !query
Expand Down Expand Up @@ -967,7 +967,7 @@ Results [2]: [key#x, buf#x]

(4) Exchange
Input [2]: [key#x, buf#x]
Arguments: hashpartitioning(key#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, 4), true, [id=#x]

(5) ObjectHashAggregate
Input [2]: [key#x, buf#x]
Expand Down Expand Up @@ -1017,7 +1017,7 @@ Results [2]: [key#x, min#x]

(5) Exchange
Input [2]: [key#x, min#x]
Arguments: hashpartitioning(key#x, 4), true, [id=#x]
Arguments: hashpartitioning(key, 4), true, [id=#x]

(6) Sort [codegen id : 2]
Input [2]: [key#x, min#x]
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/pivot.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@ PIVOT (
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
Invalid pivot column 'named_struct(course, course#x, m, m#x)'. Pivot columns must be comparable.;
Invalid pivot column 'named_struct(course, course, m, m)'. Pivot columns must be comparable.;


-- !query
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/udf/udf-pivot.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ PIVOT (
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
Invalid pivot column 'named_struct(course, course#x, m, m#x)'. Pivot columns must be comparable.;
Invalid pivot column 'named_struct(course, course, m, m)'. Pivot columns must be comparable.;


-- !query
Expand Down