Skip to content

[WIP] OperatorStateMetadata validation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericm-db wants to merge 15 commits into from
Closed

[WIP] OperatorStateMetadata validation #14

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0fd4413
feedback
ericm-db Jul 2, 2024
bbb7091
removing println, test passes
ericm-db Jul 9, 2024
efd1a09
keySchema
ericm-db Jul 10, 2024
47ea28b
rebase
ericm-db Jul 10, 2024
fbe62d7
keySchema
ericm-db Jul 10, 2024
43bf78f
rebase
ericm-db Jul 10, 2024
c3fc316
writing schema
ericm-db Jun 25, 2024
2dcd90e
purging
ericm-db Jul 11, 2024
4ba3d58
init
ericm-db Jul 15, 2024
bc24c1f
added test case and error msg
ericm-db Jul 15, 2024
e0d8ffb
it works
ericm-db Jul 16, 2024
948b2f7
it compiles lol
ericm-db Jul 16, 2024
405eb55
removing unnecessary files
ericm-db Jul 16, 2024
9688905
validation
ericm-db Jul 16, 2024
654133c
removing old code
ericm-db Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
removing println, test passes
  • Loading branch information
@ericm-db
ericm-db committed Jul 16, 2024
commit bbb7091778ed2cb813a5963015d50f16d1af6e94
6 changes: 6 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,12 @@ class IncrementalExecution(
*/
private val STATE_SCHEMA_DEFAULT_VERSION: Int = 2

/**
* This value dictates which schema format version the state schema should be written in
* for all operators other than TransformWithState.
*/
private val STATE_SCHEMA_DEFAULT_VERSION: Int = 2

/**
* See [SPARK-18339]
* Walk the optimized logical plan and replace CurrentBatchTimestamp
Expand Down
1 change: 0 additions & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -884,7 +884,6 @@ class TransformWithStateSuite extends StateStoreMetricsTest

val columnFamilySchemas = fetchColumnFamilySchemas(chkptDir.getCanonicalPath, 0)
assert(columnFamilySchemas.length == 1)

val expected = ColumnFamilySchemaV1(
"countState",
new StructType().add("key",
Expand Down