[SPARK-53490][CONNECT][SQL] Fix Protobuf conversion in observed metrics #52236
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
heyihong
changed the title
heyihong
force-pushed
the
SPARK-53490
branch
from
c83d218 to
7ba8cff
Compare
heyihong
changed the title
heyihong
force-pushed
the
SPARK-53490
branch
from
7ba8cff to
6fb749d
Compare
Contributor
Author
heyihong
force-pushed
the
SPARK-53490
branch
from
6fb749d to
e021d99
Compare
heyihong
force-pushed
the
SPARK-53490
branch
from
e021d99 to
729233e
Compare
ueshin
approved these changes
Sep 8, 2025
Member
ueshin
left a comment
ueshin
left a comment
There was a problem hiding this comment.
LGTM.
What's the plan for Python client?
heyihong
force-pushed
the
SPARK-53490
branch
from
729233e to
28ea055
Compare
Contributor
Author
I am not sure if classic/connect mode support complex data types in PySpark. I will follow up on the Python client in this ticket. |
heyihong
force-pushed
the
SPARK-53490
branch
from
28ea055 to
1ce1d4b
Compare
heyihong
force-pushed
the
SPARK-53490
branch
from
1ce1d4b to
3915e72
Compare
Contributor
|
merged to master |
huangxiaopingRD
pushed a commit
to huangxiaopingRD/spark
that referenced
this pull request
Nov 25, 2025
### What changes were proposed in this pull request?
This PR fixes a critical issue in the protobuf conversion of observed metrics in Spark Connect, specifically when dealing with complex data types like structs, arrays, and maps. The main changes include:
1. **Modified Observation class to store Row objects instead of Map[String, Any]**: Changed the internal promise type from `Promise[Map[String, Any]]` to `Promise[Row]` to preserve type information during protobuf serialization/deserialization.
2. **Enhanced protobuf conversion for complex types**:
- Added proper handling for struct types by creating `GenericRowWithSchema` objects instead of tuples
- Added support for map type conversion in `LiteralValueProtoConverter`
- Improved data type inference with a new `getDataType` method that properly handles all literal types
3. **Fixed observed metrics**: Modified the observed metrics processing to include data type information in the protobuf conversion, ensuring that complex types are properly serialized and deserialized.
### Why are the changes needed?
The previous implementation had several issues:
1. **Data type loss**: Observed metrics were losing their original data types during Protobuf conversion, causing errors
2. **Struct handling problems**: The conversion logic didn't properly handle Row objects and struct types
### Does this PR introduce _any_ user-facing change?
Yes, this PR fixes a bug that was preventing users from successfully using observed metrics with complex data types (structs, arrays, maps) in Spark Connect. Users can now:
- Use `struct()` expressions in observed metrics and receive properly typed `GenericRowWithSchema` objects
- Use `array()` expressions in observed metrics and receive properly typed arrays
- Use `map()` expressions in observed metrics and receive properly typed maps
Previously, the code below would fail.
```scala
val observation = Observation("struct")
spark
.range(10)
.observe(observation, struct(count(lit(1)).as("rows"), max("id").as("maxid")).as("struct"))
.collect()
observation.get
// Below is the error message:
"""
org.apache.spark.SparkUnsupportedOperationException: literal [10,9] not supported (yet).
org.apache.spark.sql.connect.common.LiteralValueProtoConverter$.toLiteralProtoBuilder(LiteralValueProtoConverter.scala:104)
org.apache.spark.sql.connect.common.LiteralValueProtoConverter$.toLiteralProto(LiteralValueProtoConverter.scala:203)
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution$.$anonfun$createObservedMetricsResponse$2(SparkConnectPlanExecution.scala:571)
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution$.$anonfun$createObservedMetricsResponse$2$adapted(SparkConnectPlanExecution.scala:570)
"""
```
### How was this patch tested?
`build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite -- -z SPARK-53490"`
`build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 1.5.9
Closes apache#52236 from heyihong/SPARK-53490.
Authored-by: Yihong He <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes a critical issue in the protobuf conversion of observed metrics in Spark Connect, specifically when dealing with complex data types like structs, arrays, and maps. The main changes include:
Promise[Map[String, Any]]toPromise[Row]to preserve type information during protobuf serialization/deserialization.GenericRowWithSchemaobjects instead of tuplesLiteralValueProtoConvertergetDataTypemethod that properly handles all literal typesWhy are the changes needed?
The previous implementation had several issues:
Does this PR introduce any user-facing change?
Yes, this PR fixes a bug that was preventing users from successfully using observed metrics with complex data types (structs, arrays, maps) in Spark Connect. Users can now:
struct()expressions in observed metrics and receive properly typedGenericRowWithSchemaobjectsarray()expressions in observed metrics and receive properly typed arraysmap()expressions in observed metrics and receive properly typed mapsPreviously, the code below would fail.
How was this patch tested?
build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite -- -z SPARK-53490"build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 1.5.9