Skip to content

Conversation

@srielau
Copy link
Contributor

@srielau srielau commented Apr 3, 2024

What changes were proposed in this pull request?

We changed the time parser policy in Spark 3.0.0.
The config has since defaulted to raise an exception if there is a potential conflict between teh legacy and the new policy.
Spark 4.0.0 is a good time to default to the new policy

Why are the changes needed?

Move the product forward and retire legacy behavior over time.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Run existing unit tests and verify changes.

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the SQL label Apr 3, 2024
@HyukjinKwon HyukjinKwon changed the title [SPARK-47719] change spark.sql.legacy.timeParserPolicy default to CORRECTED [SPARK-47719][SQL] Change spark.sql.legacy.timeParserPolicy default to CORRECTED Apr 3, 2024
@HyukjinKwon
Copy link
Member

and this seems a duplicate of #45853 cc @cloud-fan

@cloud-fan
Copy link
Contributor

======================================================================
FAIL [2.071s]: test_error_enrichment_jvm_stacktrace (pyspark.sql.tests.connect.test_connect_session.SparkConnectSessionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_session.py", line 137, in test_error_enrichment_jvm_stacktrace
    self.spark.sql(
AssertionError: SparkUpgradeException not raised

We need to update one pyspark test as well.

@srielau
Copy link
Contributor Author

srielau commented Apr 5, 2024

@gengliangwang @cloud-fan I think it's ready now. (one spark connect agrees test suite).

@github-actions github-actions bot added the PYTHON label Apr 5, 2024
@gengliangwang
Copy link
Member

Thanks, merging to master.
cc @dongjoon-hyun @yaooqinn who approved #45853

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants