Skip to content

Fix the way we decide if hasNext is true or not in WALReader. Unit tests... #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tdas merged 6 commits into from
Oct 4, 2014
Merged

Fix the way we decide if hasNext is true or not in WALReader. Unit tests... #13

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d08fde
Fix the way we decide if hasNext is true or not in WALReader. Unit te…
harishreedharan Oct 2, 2014
073d1f8
Use ByteBuffer in the WAL
harishreedharan Oct 3, 2014
164bd14
Rewind buffer in call cases
harishreedharan Oct 3, 2014
d8f14cd
Consolidate all tests into one class. Take Hadoop config as a param t…
harishreedharan Oct 3, 2014
a2457e4
Wrap getting the hflush in a method.
harishreedharan Oct 3, 2014
bbfeae1
Use sync method if hflush is not available.
harishreedharan Oct 3, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Wrap getting the hflush in a method.
  • Loading branch information
@harishreedharan
harishreedharan committed Oct 3, 2014
commit a2457e4ede56a8025c8a1c4c08d3513bd4eea2c6
17 changes: 10 additions & 7 deletions streaming/src/main/scala/org/apache/spark/streaming/storage/WriteAheadLogWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package org.apache.spark.streaming.storage

import java.io.Closeable
import java.lang.reflect.Method
import java.nio.ByteBuffer

import org.apache.hadoop.conf.Configuration
Expand All @@ -26,13 +27,7 @@ private[streaming] class WriteAheadLogWriter(path: String, conf: Configuration)
private val stream = HdfsUtils.getOutputStream(path, conf)
private var nextOffset = stream.getPos
private var closed = false
private val hflushMethod = {
try {
Some(classOf[FSDataOutputStream].getMethod("hflush"))
} catch {
case e: Exception => None
}
}
private val hflushMethod = getHflushMethod()

// Data is always written as:
// - Length - Long
Expand Down Expand Up @@ -65,6 +60,14 @@ private[streaming] class WriteAheadLogWriter(path: String, conf: Configuration)
hflushMethod.foreach(_.invoke(stream))
}

private def getHflushMethod(): Option[Method] = {
try {
Some(classOf[FSDataOutputStream].getMethod("hflush"))
} catch {
case e: Exception => None
}
}

private def assertOpen() {
HdfsUtils.checkState(!closed, "Stream is closed. Create a new Writer to write to file.")
}
Expand Down