Skip to content

[WIP] OperatorStateMetadata validation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericm-db wants to merge 15 commits into from
Closed

[WIP] OperatorStateMetadata validation #14

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0fd4413
feedback
ericm-db Jul 2, 2024
bbb7091
removing println, test passes
ericm-db Jul 9, 2024
efd1a09
keySchema
ericm-db Jul 10, 2024
47ea28b
rebase
ericm-db Jul 10, 2024
fbe62d7
keySchema
ericm-db Jul 10, 2024
43bf78f
rebase
ericm-db Jul 10, 2024
c3fc316
writing schema
ericm-db Jun 25, 2024
2dcd90e
purging
ericm-db Jul 11, 2024
4ba3d58
init
ericm-db Jul 15, 2024
bc24c1f
added test case and error msg
ericm-db Jul 15, 2024
e0d8ffb
it works
ericm-db Jul 16, 2024
948b2f7
it compiles lol
ericm-db Jul 16, 2024
405eb55
removing unnecessary files
ericm-db Jul 16, 2024
9688905
validation
ericm-db Jul 16, 2024
654133c
removing old code
ericm-db Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
removing unnecessary files
  • Loading branch information
@ericm-db
ericm-db committed Jul 16, 2024
commit 405eb555957c22f41d4b5719493ef8ca569e8ca3
100 changes: 0 additions & 100 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateSchemaV3File.scala
View file
Open in desktop

This file was deleted.

3 changes: 1 addition & 2 deletions ...core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,7 @@ case class TransformWithStateExec(
initialStateDataAttrs: Seq[Attribute],
initialStateDeserializer: Expression,
initialState: SparkPlan)
extends BinaryExecNode with StateStoreWriter with WatermarkSupport with ObjectProducerExec
with Logging {
extends BinaryExecNode with StateStoreWriter with WatermarkSupport with ObjectProducerExec {

override def shortName: String = "transformWithStateExec"

Expand Down