Skip to content

Conversation

@maropu
Copy link
Member

@maropu maropu commented Aug 2, 2018

What changes were proposed in this pull request?

In the current master, toString throws an exception when RelationalGroupedDataset has unresolved expressions;

scala> spark.range(0, 10).groupBy("id")
res4: org.apache.spark.sql.RelationalGroupedDataset = RelationalGroupedDataset: [grouping expressions: [id: bigint], value: [id: bigint], type: GroupBy]

scala> spark.range(0, 10).groupBy('id)
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'id
  at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:105)
  at org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:474)
  at org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:473)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.AbstractTraversable.map(Traversable.scala:104)
  at org.apache.spark.sql.RelationalGroupedDataset.toString(RelationalGroupedDataset.scala:473)
  at scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)
  at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)
  at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)

This pr fixed code to handle the unresolved case in RelationalGroupedDataset.toString.

Closes #21752

How was this patch tested?

Added tests in DataFrameAggregateSuite.

@maropu
Copy link
Member Author

maropu commented Aug 2, 2018

@gatorsmile KeyValueGroupedDataset has the same issue? It seems there is no chance for KeyValueGroupedDataset to have unresolved exprs.
#21752 (comment)

@SparkQA
Copy link

SparkQA commented Aug 2, 2018

Test build #93962 has finished for PR 21964 at commit c4e7490.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Aug 2, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Aug 2, 2018

Test build #93975 has finished for PR 21964 at commit c4e7490.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Aug 2, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Aug 2, 2018

Test build #93995 has finished for PR 21964 at commit c4e7490.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Aug 2, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Aug 2, 2018

Test build #94007 has finished for PR 21964 at commit c4e7490.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member Author

maropu commented Aug 3, 2018

@gatorsmile ping

Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks! Merged to master.

@asfgit asfgit closed this in b0d6967 Aug 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants