Skip to content

[SPARK-22187][SS] Update unsaferow format for saved state in flatMapGroupsWithState to allow timeouts with deleted state #21739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tdas wants to merge 11 commits into from
Closed

[SPARK-22187][SS] Update unsaferow format for saved state in flatMapGroupsWithState to allow timeouts with deleted state #21739

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ef509c8
Partial implementation
tdas Oct 5, 2017
976a7ea
Fixed and added tests
tdas Jul 9, 2018
cfc3f68
Refactored
tdas Jul 9, 2018
9525484
Minor refactoring
tdas Jul 9, 2018
c9f600b
Added conf and multi-version tests in FlatMapGroupsWithStateSuite
tdas Jul 10, 2018
dbd08ca
Merge remote-tracking branch 'apache-github/master' into SPARK-22187-1
tdas Jul 11, 2018
05e3acf
Use version 1 when recovering from state
tdas Jul 11, 2018
dcf9616
Revert few unnecessary changes
tdas Jul 11, 2018
3abb5e2
minor changes
tdas Jul 11, 2018
58ce6e1
Merge remote-tracking branch 'apache-github/master' into SPARK-22187-1
tdas Jul 19, 2018
c262e87
Added docs and unit tests to verify format
tdas Jul 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'apache-github/master' into SPARK-22187-1
  • Loading branch information
@tdas
tdas committed Jul 19, 2018
commit 58ce6e13741b39a9b724378c98561a3acba9e596
2 changes: 1 addition & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -714,7 +714,7 @@ trait ComplexTypeMergingExpression extends Expression {
inputTypesForMerging.nonEmpty,
"The collection of input data types must not be empty.")
require(
areInputTypesForMergingEqual,
TypeCoercion.haveSameType(inputTypesForMerging),
"All input types must be the same except nullable, containsNull, valueContainsNull flags." +
s" The input types found are\n\t${inputTypesForMerging.mkString("\n\t")}")
inputTypesForMerging.reduceLeft(TypeCoercion.findCommonTypeDifferentOnlyInNullFlags(_, _).get)
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.