Skip to content

Conversation

@khalidmammadov
Copy link
Contributor

What changes were proposed in this pull request?

Incorrect parameter error is used for CANNOT_PARSE_DATATYPE error class in PySpark code. It expects msg. Due to this the error produced during usage is incorrect:

Example:

StructField.fromJson({"name": "col1", "type": "NewType", "nullable": "True", "metadata": None}) 

Error:

assert set(message_parameters_from_template) == set(message_parameters), (
AssertionError: Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': 'NewType'} 

Why are the changes needed?

Bug fix

Does this PR introduce any user-facing change?

Yes, now correct error would be printed into console:

raise PySparkValueError(
pyspark.errors.exceptions.base.PySparkValueError: [CANNOT_PARSE_DATATYPE] Unable to parse datatype. NewType.

How was this patch tested?

Added unit test and tested locally.

Was this patch authored or co-authored using generative AI tooling?

No

@khalidmammadov
Copy link
Contributor Author

looks like it was already fixed. closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant