Skip to content

[WIP] OperatorStateMetadata validation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericm-db wants to merge 15 commits into from
Closed

[WIP] OperatorStateMetadata validation #14

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0fd4413
feedback
ericm-db Jul 2, 2024
bbb7091
removing println, test passes
ericm-db Jul 9, 2024
efd1a09
keySchema
ericm-db Jul 10, 2024
47ea28b
rebase
ericm-db Jul 10, 2024
fbe62d7
keySchema
ericm-db Jul 10, 2024
43bf78f
rebase
ericm-db Jul 10, 2024
c3fc316
writing schema
ericm-db Jun 25, 2024
2dcd90e
purging
ericm-db Jul 11, 2024
4ba3d58
init
ericm-db Jul 15, 2024
bc24c1f
added test case and error msg
ericm-db Jul 15, 2024
e0d8ffb
it works
ericm-db Jul 16, 2024
948b2f7
it compiles lol
ericm-db Jul 16, 2024
405eb55
removing unnecessary files
ericm-db Jul 16, 2024
9688905
validation
ericm-db Jul 16, 2024
654133c
removing old code
ericm-db Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
removing old code
  • Loading branch information
@ericm-db
ericm-db committed Jul 16, 2024
commit 654133c3c6729aaf021f548ca8c85a61bf48dd81
10 changes: 0 additions & 10 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImplWithTTL.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,6 @@ import org.apache.spark.sql.execution.streaming.state.{PrefixKeyScanStateEncoder
import org.apache.spark.sql.streaming.{MapState, TTLConfig}
import org.apache.spark.util.NextIterator

object MapStateImplWithTTL {
def columnFamilySchema(stateName: String): ColumnFamilySchemaV1 = {
new ColumnFamilySchemaV1(
stateName,
COMPOSITE_KEY_ROW_SCHEMA,
VALUE_ROW_SCHEMA_WITH_TTL,
PrefixKeyScanStateEncoderSpec(COMPOSITE_KEY_ROW_SCHEMA, 1), false)
}
}

/**
* Class that provides a concrete implementation for map state associated with state
* variables (with ttl expiration support) used in the streaming transformWithState operator.
Expand Down