Skip to content

[WIP] OperatorStateMetadata validation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericm-db wants to merge 15 commits into from
Closed

[WIP] OperatorStateMetadata validation #14

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0fd4413
feedback
ericm-db Jul 2, 2024
bbb7091
removing println, test passes
ericm-db Jul 9, 2024
efd1a09
keySchema
ericm-db Jul 10, 2024
47ea28b
rebase
ericm-db Jul 10, 2024
fbe62d7
keySchema
ericm-db Jul 10, 2024
43bf78f
rebase
ericm-db Jul 10, 2024
c3fc316
writing schema
ericm-db Jun 25, 2024
2dcd90e
purging
ericm-db Jul 11, 2024
4ba3d58
init
ericm-db Jul 15, 2024
bc24c1f
added test case and error msg
ericm-db Jul 15, 2024
e0d8ffb
it works
ericm-db Jul 16, 2024
948b2f7
it compiles lol
ericm-db Jul 16, 2024
405eb55
removing unnecessary files
ericm-db Jul 16, 2024
9688905
validation
ericm-db Jul 16, 2024
654133c
removing old code
ericm-db Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rebase
  • Loading branch information
@ericm-db
ericm-db committed Jul 16, 2024
commit 43bf78fd82db85867a5213981e82cb2f33d9dcef
3 changes: 1 addition & 2 deletions ...ore/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaV3File.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ import scala.io.{Source => IOSource}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path

import org.apache.spark.internal.Logging
import org.apache.spark.sql.execution.streaming.CheckpointFileManager
import org.apache.spark.sql.execution.streaming.MetadataVersionUtil.validateVersion

Expand All @@ -40,7 +39,7 @@ import org.apache.spark.sql.execution.streaming.MetadataVersionUtil.validateVers
*/
class StateSchemaV3File(
hadoopConf: Configuration,
path: String) extends Logging {
path: String) {

val metadataPath = new Path(path)

Expand Down
5 changes: 3 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ import org.apache.spark.sql.catalyst.util.UnsafeRowUtils
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
import org.apache.spark.sql.execution.streaming.StatefulOperatorStateInfo
import org.apache.spark.sql.execution.streaming.TransformWithStateKeyValueRowSchema.{COMPOSITE_KEY_ROW_SCHEMA, KEY_ROW_SCHEMA}
import org.apache.spark.sql.types.StructType
import org.apache.spark.util.{NextIterator, ThreadUtils, Utils}

Expand Down Expand Up @@ -292,14 +293,14 @@ object KeyStateEncoderSpec {
// match on type
m("keyStateEncoderType").asInstanceOf[String] match {
case "NoPrefixKeyStateEncoderSpec" =>
NoPrefixKeyStateEncoderSpec(keySchema)
NoPrefixKeyStateEncoderSpec(KEY_ROW_SCHEMA)
case "RangeKeyScanStateEncoderSpec" =>
val orderingOrdinals = m("orderingOrdinals").
asInstanceOf[List[_]].map(_.asInstanceOf[BigInt].toInt)
RangeKeyScanStateEncoderSpec(keySchema, orderingOrdinals)
case "PrefixKeyScanStateEncoderSpec" =>
val numColsPrefixKey = m("numColsPrefixKey").asInstanceOf[BigInt]
PrefixKeyScanStateEncoderSpec(keySchema, numColsPrefixKey.toInt)
PrefixKeyScanStateEncoderSpec(COMPOSITE_KEY_ROW_SCHEMA, numColsPrefixKey.toInt)
}
}
}
Expand Down