Skip to content

[SPARK-28120][SS] Rocksdb state storage implementation #24922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
itsvikramagr wants to merge 28 commits into from
Closed

[SPARK-28120][SS] Rocksdb state storage implementation #24922

Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f59d881
Rocksdb state storage implementation
Jun 20, 2019
292befe
Add Apache License
Jun 20, 2019
05568ee
Fix indentation and scalastyle errors
Jun 25, 2019
3f5f6b2
remove unwanted lines
Jun 25, 2019
f0f2f8d
Fix review comments
Jun 25, 2019
827ace4
update hadoop manifest file
Jun 25, 2019
db1ed2b
make local directory a configurable option
Jun 26, 2019
3ad88eb
Review Comments Fixes
Jul 8, 2019
41a688d
More Review comments fixes
Jul 8, 2019
a484d63
Address Review Comments and fix memory leaks
Jul 31, 2019
35de011
resolve merge conflicts in spark-deps-hadoop-2.7
Jul 31, 2019
f983d78
minor change
Jul 31, 2019
d41560a
fix dependencies issue
Jul 31, 2019
ebaea37
Fix documentation issues
Jul 31, 2019
818f716
minor changes
Jul 31, 2019
603958b
List directories instead of using a loop to clean up checkpointed roc…
Aug 7, 2019
562f755
Address Review Comments
Aug 28, 2019
4f42068
Unit tests for FileUtility
Aug 28, 2019
7d4d5d1
Update spark dependencies
Aug 28, 2019
0b129f3
Revert POM dependency change
Aug 28, 2019
c38bd6c
Add dependencies
Aug 28, 2019
fcf2a86
Fix fileutility unit test
Aug 28, 2019
fb86f0d
Minor Fixes in FileUtilitySuite
Aug 29, 2019
4544abc
Dummy commit to trigger the build
Aug 29, 2019
8f50519
sync with master
Sep 24, 2019
b3ef8ea
create RocksDbStateStoreConf for rocksdb related configs
Sep 24, 2019
417de56
cache should be singleton
Sep 24, 2019
45e0d05
Add ConfigBuilder for Rocksdb configuration
Sep 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor change
  • Loading branch information
Vikram Agrawal committed Jul 31, 2019
commit f983d7831d07f9e0046345d28d645da69b783605
2 changes: 1 addition & 1 deletion ...main/scala/org/apache/spark/sql/execution/streaming/state/RocksDbStateStoreProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ private[sql] class RocksDbStateStoreProvider extends StateStoreProvider with Log
}

private def getLastCommittedVersion(): Long = {
val f = new File(rocksDbPath, "commit")
val f = new File(rocksDbPath, RocksDbInstance.COMMIT_FILE_NAME)
if (f.exists()) {
try {
val fileContents = scala.io.Source.fromFile(f.getAbsolutePath).getLines.mkString
Expand Down