Skip to content

Initial HDFS Readers and Writers implementation. #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tdas merged 10 commits into from
Oct 2, 2014
Merged

Initial HDFS Readers and Writers implementation. #12

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e4e6f93
Basic implementation of HDFS WAL.
harishreedharan Oct 1, 2014
f6d3a9d
Make most methods in writer and reader thread-safe.
harishreedharan Oct 1, 2014
c6ffa46
Add a method to close files
harishreedharan Oct 1, 2014
ea6f1b4
Add preconditions checks to ensure that the writer or reader has not …
harishreedharan Oct 1, 2014
faff282
Making FileSegment a case class
harishreedharan Oct 1, 2014
46e03ab
Making all HDFS Wal classes private to streaming
harishreedharan Oct 1, 2014
a810b0a
Make close methods synchronized, to ensure close does not get called …
harishreedharan Oct 1, 2014
fc53eac
Make the Sequential reader an Iterator. Rename all classes. Mark all …
harishreedharan Oct 1, 2014
9dfbd66
Make all variables private
harishreedharan Oct 2, 2014
d86a518
Minor changes to definitions of methods
harishreedharan Oct 2, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add a method to close files
  • Loading branch information
@harishreedharan
harishreedharan committed Oct 1, 2014
commit c6ffa465de0dc15b8dfb7c8183ea9d9501a405ee
4 changes: 4 additions & 0 deletions streaming/src/main/scala/org/apache/spark/streaming/storage/HdfsSequentialReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,8 @@ class HdfsSequentialReader(val path: String) {
buffer
}
}

def close() {
instream.close()
}
}
4 changes: 4 additions & 0 deletions streaming/src/main/scala/org/apache/spark/streaming/storage/HdfsWalRandomReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,9 @@ class HdfsWalRandomReader(val path: String) {
buffer
}
}

def close() {
instream.close()
}
}

4 changes: 4 additions & 0 deletions streaming/src/main/scala/org/apache/spark/streaming/storage/HdfsWalWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,8 @@ class HdfsWalWriter(val path: String) {
segment
}
}

def close(): Unit = {
stream.close()
}
}