[SPARK-24544][SQL] Print actual failure cause when look up function failed #21790

caneGuy · 2018-07-17T10:08:22Z

What changes were proposed in this pull request?

When we operate as below:
0: jdbc:hive2://xxx/> create function funnel_analysis as 'com.xxx.hive.extend.udf.UapFunnelAnalysis';

0: jdbc:hive2://xxx/> select funnel_analysis(1,",",1,''); Error: org.apache.spark.sql.AnalysisException: Undefined function: 'funnel_analysis'. This function is neither a registered temporary function nor a permanent function registered in the database 'xxx'.; line 1 pos 7 (state=,code=0)

0: jdbc:hive2://xxx/> describe function funnel_analysis; +-----------------------------------------------------------+--+ | function_desc | +-----------------------------------------------------------+--+ | Function: xxx.funnel_analysis | | Class: com.xxx.hive.extend.udf.UapFunnelAnalysis | | Usage: N/A. | +-----------------------------------------------------------+--+
We can see describe funtion will get right information,but when we actually use this funtion,we will get an undefined exception.
Which is really misleading,the real cause is below:
No handler for Hive UDF 'com.xxx.xxx.hive.extend.udf.UapFunnelAnalysis': java.lang.IllegalStateException: Should not be called directly; at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72) at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:204) at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:204) at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:212) at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:212)
This patch print the actual failure for quick debugging.

How was this patch tested?

UT

…ailed

caneGuy · 2018-07-17T10:09:29Z

@maropu Could you help review this?Thanks
since #21552 i used git merge so reopen this.

SparkQA · 2018-07-17T13:22:07Z

Test build #93167 has finished for PR 21790 at commit 690035a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class NoSuchFunctionException(db: String, func: String, rootCause: Option[String] = None)

caneGuy · 2018-08-28T06:50:19Z

retest this please

maropu · 2018-08-28T06:57:52Z

sure, I'll check later.

SparkQA · 2018-08-28T07:05:02Z

Test build #95330 has finished for PR 21790 at commit 690035a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class NoSuchFunctionException(db: String, func: String, rootCause: Option[String] = None)

maropu · 2018-08-30T11:24:15Z

retest this please

maropu · 2018-08-30T11:26:01Z

@caneGuy Can you fix the title?

maropu · 2018-08-30T11:26:45Z

Also, you need to add tests.

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala

SparkQA · 2018-08-30T14:36:58Z

Test build #95465 has finished for PR 21790 at commit 690035a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class NoSuchFunctionException(db: String, func: String, rootCause: Option[String] = None)

caneGuy · 2018-10-22T14:07:35Z

I will fix the test and add unit test as soon as possible @maropu
I am too busy last month,sorry for the late reply.
Thanks for your comments and your precious time again!

caneGuy · 2018-12-19T11:39:32Z

Sorry to bother you @maropu .I tried to add some test case,but i found SessionCatalogSuite has test the look up logic.
Any suggestion to add some more test case?

SparkQA · 2018-12-19T21:58:57Z

Test build #100310 has finished for PR 21790 at commit e1866d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class NoSuchFunctionException(db: String, func: String, cause: Option[Throwable] = None)

maropu · 2018-12-27T03:49:42Z

I think we need tests for the changes this pr added.

caneGuy · 2018-12-27T11:34:14Z

@maropu Thanks for your time.I add a test case in SessionCatalogSuite

srowen · 2018-12-27T14:30:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala

-    s"a permanent function registered in the database '$db'.")
+     s"Undefined function: '$func'. This function is neither a registered temporary function nor " +
+      s"a permanent function registered in the database '$db'." +
+      s"${cause.map(th => s"Exception thrown during look up:" +


This is too complex for string interpolation. But why not just set the exception's cause?

@srowen Thanks.IIUC.Since some times the cause may be undefined.For that case, we do not need to print Exception thrown during look up.

I mean, why not set the cause on AnalysisException? the cause may be null if the Option is None, that's fine.

Update @srowen Thanks!

SparkQA · 2018-12-27T15:42:34Z

Test build #100475 has finished for PR 21790 at commit ec208e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-29T08:05:01Z

Test build #100524 has finished for PR 21790 at commit 95ba6e7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-29T08:05:01Z

Test build #100522 has finished for PR 21790 at commit 26aeb58.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-29T12:59:28Z

Test build #100529 has finished for PR 21790 at commit 1bff63c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2018-12-29T13:03:35Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala

          // built-in function.
          // Hive is case insensitive.
          val functionName = funcName.unquotedString.toLowerCase(Locale.ROOT)
+          logWarning(s"Encounter a failure during looking up function:" +


Nit: Encounter -> Encountered, and the first string doesn't need interpolation

Thanks @srowen Updated!

SparkQA · 2018-12-30T17:04:09Z

Test build #100558 has finished for PR 21790 at commit 820faf2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2018-12-31T14:22:32Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala

+
+  test("SPARK-24544: test print actual failure cause when look up function failed") {
+    withBasicCatalog { catalog =>
+      intercept[NoSuchFunctionException] {


Does this test anything about the cause of the exception? maybe get the cause and assert about its message?

SparkQA · 2019-01-01T03:00:39Z

Test build #100600 has finished for PR 21790 at commit 6d3bd19.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-01T05:56:55Z

Test build #100603 has finished for PR 21790 at commit 97493c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-01T06:07:17Z

Test build #100604 has finished for PR 21790 at commit d1d4adf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-01-01T15:13:32Z

Merged to master

…ailed ## What changes were proposed in this pull request? When we operate as below: ` 0: jdbc:hive2://xxx/> create function funnel_analysis as 'com.xxx.hive.extend.udf.UapFunnelAnalysis'; ` ` 0: jdbc:hive2://xxx/> select funnel_analysis(1,",",1,''); Error: org.apache.spark.sql.AnalysisException: Undefined function: 'funnel_analysis'. This function is neither a registered temporary function nor a permanent function registered in the database 'xxx'.; line 1 pos 7 (state=,code=0) ` ` 0: jdbc:hive2://xxx/> describe function funnel_analysis; +-----------------------------------------------------------+--+ | function_desc | +-----------------------------------------------------------+--+ | Function: xxx.funnel_analysis | | Class: com.xxx.hive.extend.udf.UapFunnelAnalysis | | Usage: N/A. | +-----------------------------------------------------------+--+ ` We can see describe funtion will get right information,but when we actually use this funtion,we will get an undefined exception. Which is really misleading,the real cause is below: ` No handler for Hive UDF 'com.xxx.xxx.hive.extend.udf.UapFunnelAnalysis': java.lang.IllegalStateException: Should not be called directly; at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72) at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:204) at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:204) at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:212) at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:212) ` This patch print the actual failure for quick debugging. ## How was this patch tested? UT Closes apache#21790 from caneGuy/zhoukang/print-warning1. Authored-by: zhoukang <[email protected]> Signed-off-by: Sean Owen <[email protected]>

dongjoon-hyun · 2019-06-03T05:02:16Z

Hi, All.
This seems to cause a regression to show misleading warnings always at first invocation for all Hive function. Hive fallback lookup should not be warned. It's a normal process in function lookups. Since this bug is not exposed to the release, I'll make a followup on master branch.

srowen · 2019-06-03T11:45:21Z

OK, if it's pretty common after all, this can be reverted, thank you

…n look up function failed` ### What changes were proposed in this pull request? This reverts #21790 because it's no longer needed. It kept the original error from Hive when Spark loads builtin functions from Hive, which no longer happens as Spark has implemented all builtin functions natively. ### Why are the changes needed? code cleanup ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing tests Closes #37896 from cloud-fan/error. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…n look up function failed` ### What changes were proposed in this pull request? This reverts apache#21790 because it's no longer needed. It kept the original error from Hive when Spark loads builtin functions from Hive, which no longer happens as Spark has implemented all builtin functions natively. ### Why are the changes needed? code cleanup ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing tests Closes apache#37896 from cloud-fan/error. Lead-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

[SPARK-24544][SQL] Print actual failure cause when look up function f…

690035a

…ailed

maropu reviewed Aug 30, 2018

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala Outdated Show resolved Hide resolved

maropu reviewed Aug 30, 2018

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala Outdated Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala Outdated Show resolved Hide resolved

caneGuy changed the title ~~[SPARK-24544][SQL] Print actual failure cause when look up function f…~~ [SPARK-24544][SQL] Print actual failure cause when look up function failed Dec 19, 2018

Refine comment

e1866d1

Add unit test

ec208e0

srowen reviewed Dec 27, 2018

View reviewed changes

caneGuy added 2 commits December 29, 2018 12:33

Refine comment

26aeb58

Update style

95ba6e7

Remove useless import

1bff63c

srowen reviewed Dec 29, 2018

View reviewed changes

Refine comment

820faf2

srowen approved these changes Dec 30, 2018

View reviewed changes

srowen reviewed Dec 31, 2018

View reviewed changes

caneGuy added 4 commits January 1, 2019 08:46

Refine

240f3ab

Refine

6d3bd19

Refine

97493c9

Add comment

d1d4adf

srowen closed this in 2bf4d97 Jan 1, 2019

caneGuy deleted the zhoukang/print-warning1 branch January 14, 2019 13:49

dongjoon-hyun mentioned this pull request Jun 3, 2019

[SPARK-24544][SQL][FOLLOWUP] Remove a wrong warning on Hive fallback lookup #24773

Closed

cloud-fan mentioned this pull request Sep 15, 2022

[SPARK-40482][SQL] Revert SPARK-24544 Print actual failure cause when look up function failed #37896

Closed

[SPARK-24544][SQL] Print actual failure cause when look up function failed #21790

[SPARK-24544][SQL] Print actual failure cause when look up function failed #21790

Uh oh!

Conversation

caneGuy commented Jul 17, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

caneGuy commented Jul 17, 2018

Uh oh!

SparkQA commented Jul 17, 2018

Uh oh!

caneGuy commented Aug 28, 2018

Uh oh!

maropu commented Aug 28, 2018

Uh oh!

SparkQA commented Aug 28, 2018

Uh oh!

maropu commented Aug 30, 2018

Uh oh!

maropu commented Aug 30, 2018

Uh oh!

maropu commented Aug 30, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

caneGuy commented Oct 22, 2018

Uh oh!

caneGuy commented Dec 19, 2018

Uh oh!

SparkQA commented Dec 19, 2018

Uh oh!

maropu commented Dec 27, 2018

Uh oh!

caneGuy commented Dec 27, 2018

Uh oh!

srowen Dec 27, 2018

Choose a reason for hiding this comment

Uh oh!

caneGuy Dec 28, 2018

Choose a reason for hiding this comment

Uh oh!

srowen Dec 28, 2018

Choose a reason for hiding this comment

Uh oh!

caneGuy Dec 29, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 27, 2018

Uh oh!

SparkQA commented Dec 29, 2018

Uh oh!

SparkQA commented Dec 29, 2018

Uh oh!

SparkQA commented Dec 29, 2018

Uh oh!

srowen Dec 29, 2018

Choose a reason for hiding this comment

Uh oh!

caneGuy Dec 30, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 30, 2018

Uh oh!

srowen Dec 31, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 1, 2019

Uh oh!

SparkQA commented Jan 1, 2019

Uh oh!

SparkQA commented Jan 1, 2019

Uh oh!

srowen commented Jan 1, 2019

Uh oh!

dongjoon-hyun commented Jun 3, 2019

Uh oh!