Skip to content

[SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog #23850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HeartSaVioR wants to merge 4 commits into from
Closed

[SPARK-26949][SS] Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog #23850

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9c5fa64
[SPARK-26949][SS] Prevent 'purge' to remove needed batch files in Com…
HeartSaVioR Feb 21, 2019
a51564f
Fix a silly mistake
HeartSaVioR Feb 21, 2019
ba62794
Refine sentence on exception
HeartSaVioR Jun 12, 2019
f8adab3
Fix UT
HeartSaVioR Jun 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix a silly mistake
  • Loading branch information
@HeartSaVioR
HeartSaVioR committed Feb 21, 2019
commit a51564f1ad08690ee8971ef12d938d479ed52f77
2 changes: 1 addition & 1 deletion ...c/test/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLogSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ class CompactibleFileStreamLogSuite extends SparkFunSuite with SharedSQLContext
val exc = intercept[UnsupportedOperationException] {
compactibleLog.purge(2)
}
assert(exc.getMessage.contains("breaks internal state of CompactibleFileStreamLog"))
assert(exc.getMessage.contains("break internal state of CompactibleFileStreamLog"))

// Below line would fail with IllegalStateException if we don't prevent purge:
// - purge(2) would delete batch 0 and 1 which batch 1 is compaction batch
Expand Down