[SPARK-47696][SQL] try_to_timestamp should handle SparkUpgradeException

[gengliangwang]

What changes were proposed in this pull request?

Currently, try_to_timestamp will throw an exception on legacy timestamp input.

> SELECT try_to_timestamp('2016-12-1', 'yyyy-MM-dd')

org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0:
Fail to parse '2016-12-1' in the new parser.
You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid datetime string. SQLSTATE: 42K0B 

It should return null instead of error.

Why are the changes needed?

Fix a bug in function try_to_timestamp

Does this PR introduce any user-facing change?

Yes, this PR introduces a bug fix: try_to_timestamp will return null instead of throwing an error on legacy timestamp string input.

How was this patch tested?

New UT

Was this patch authored or co-authored using generative AI tooling?

No

[cloud-fan]

shall we simply update ToTimestamp#eval and codegen to catch SparkUpgradeException additionally? I'm a bit worried about increasing the scope and swallow all errors.

[gengliangwang]

So in ToTimestamp, there is supposed to be a SparkUpgradeException even when failOnErroris false.
To avoid swallowing all errors, we can add another flag isTryFunction besides failOnError. It will make the code more complicated though.

[cloud-fan]

so we need to maintain 3 behaviors:

1. try_to_timestamp never fails
2. to_timestamp ansi mode always fail with invalid input
3. to_timestamp non-ansi mode only fails with SparkUpgradeException

Do I understand it correctly?

[cloud-fan]

I'm not sure why try_to_timestamp need a different behavior with to_timestamp non-ansi mode.

[gengliangwang]

I'm not sure why try_to_timestamp need a different behavior with to_timestamp non-ansi mode.

The try_* functions are supposed to return null on invalid inputs, instead of compatible with non-ansi mode. cc @srielau on this one.

[cloud-fan]

This upgrade issue seems special and we should always throw it to provide upgrade instructions. How about #45853 ?

[gengliangwang]

@cloud-fan after #45853, users will face this bug when setting the timeParserPolicy as legacy.

[cloud-fan]

We can argue this is not a bug as the upgrade exception must be thrown.

[gengliangwang]

@cloud-fan ok, let me close this one for now.