Skip to content

Fix the way we decide if hasNext is true or not in WALReader. Unit tests... #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tdas merged 6 commits into from
Oct 4, 2014
Merged

Fix the way we decide if hasNext is true or not in WALReader. Unit tests... #13

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d08fde
Fix the way we decide if hasNext is true or not in WALReader. Unit te…
harishreedharan Oct 2, 2014
073d1f8
Use ByteBuffer in the WAL
harishreedharan Oct 3, 2014
164bd14
Rewind buffer in call cases
harishreedharan Oct 3, 2014
d8f14cd
Consolidate all tests into one class. Take Hadoop config as a param t…
harishreedharan Oct 3, 2014
a2457e4
Wrap getting the hflush in a method.
harishreedharan Oct 3, 2014
bbfeae1
Use sync method if hflush is not available.
harishreedharan Oct 3, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rewind buffer in call cases
  • Loading branch information
@harishreedharan
harishreedharan committed Oct 3, 2014
commit 164bd14022d3fbd998d85810c5b8f845a9041fac
4 changes: 2 additions & 2 deletions streaming/src/main/scala/org/apache/spark/streaming/storage/WriteAheadLogWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ private[streaming] class WriteAheadLogWriter(path: String) extends Closeable {
private var closed = false
private val hflushMethod = {
Copy link
Owner

@tdas tdas Oct 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would wrap this method in hflush stuff as a function from cleanliness, so that we dont have to do hflushMethod.foreach , just call flush()

Copy link
Author

@harishreedharan harishreedharan Oct 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrapped the call in a different method. I don't want to do a look up every time we call hflush. Doing that on initialization saves a bunch of reflection lookups. For the sake of keeping it a val, I wrapped the lookup in a different method, which is called at the time of initialization

try {
Some(classOf[FSDataOutputStream].getMethod("hflush", new Array[Class[Object]](0): _*))
Some(classOf[FSDataOutputStream].getMethod("hflush"))
} catch {
case e: Exception => None
}
Expand All @@ -38,14 +38,14 @@ private[streaming] class WriteAheadLogWriter(path: String) extends Closeable {
// - Data - of length = Length
def write(data: ByteBuffer): FileSegment = synchronized {
assertOpen()
data.rewind() // Rewind to ensure all data in the buffer is retrieved
val lengthToWrite = data.remaining()
val segment = new FileSegment(path, nextOffset, lengthToWrite)
stream.writeInt(lengthToWrite)
if (data.hasArray) {
stream.write(data.array())
} else {
// If the buffer is not backed by an array we need to copy the data to an array
data.rewind() // Rewind to ensure all data in the buffer is retrieved
val dataArray = new Array[Byte](lengthToWrite)
Copy link
Owner

@tdas tdas Oct 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We dont want to copy the data multiple times! Why not just write off the data one byte at a time. That should be faster than doing two passes on the data, and allocating memory.

Copy link
Owner

@tdas tdas Oct 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

      while(byteBuffer.hasRemaining) {
        stream.writeByte(byteBuffer.get())
      }

Copy link
Owner

@tdas tdas Oct 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, never mind. Spoke to my colleagues and writing byte at a time is slower than doing extra memcopy. So its fine as is.

data.get(dataArray)
stream.write(dataArray)
Expand Down