Skip to content

[SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache #18730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
caneGuy wants to merge 19 commits into from
Closed

[SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache #18730

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7cbadc5
[SPARK][CORE] Slice write by channel
caneMi Jul 25, 2017
fc91f96
Fix code style
caneGuy Aug 23, 2017
bab91db
Refine to avoid underflow
caneGuy Aug 24, 2017
72aef67
Fix no ending loop
caneGuy Aug 24, 2017
4789772
Remove useless unit test
caneGuy Aug 24, 2017
aeabe1d
Delete unused package
caneGuy Aug 24, 2017
ca77b51
Add benchmark testing with Utils.BenchMark
caneGuy Aug 24, 2017
e84a6d7
Refine code style
caneGuy Aug 24, 2017
d708142
Update as comment replace writeFully with slice logic
caneGuy Aug 24, 2017
33a2796
Remove unused package
caneGuy Aug 24, 2017
ab384d4
No need call toInt
caneGuy Aug 24, 2017
b669351
Only need to use bytes.remaining()
caneGuy Aug 24, 2017
717f886
Use config for NIO_BUFFER_LIMIT
caneGuy Aug 24, 2017
9d3004d
Use BUFFER_WRITE_CHUNK_SIZE
caneGuy Aug 24, 2017
e48f44f
Update as commented
caneGuy Aug 24, 2017
fc184aa
Fix build error
caneGuy Aug 24, 2017
f1d67a3
Fix compile error and refine scala style
caneGuy Aug 24, 2017
14ca824
Need to convert to int to fix compile error
caneGuy Aug 25, 2017
a02575e
Avoid per-loop type cast
caneGuy Aug 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove useless unit test
  • Loading branch information
@caneGuy
caneGuy committed Aug 24, 2017
commit 478977293aadb9383740eabbaee23a43cc64b062
19 changes: 1 addition & 18 deletions core/src/test/scala/org/apache/spark/io/ChunkedByteBufferSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ package org.apache.spark.io
import java.nio.ByteBuffer

import com.google.common.io.ByteStreams

import org.apache.spark.SparkFunSuite
import org.apache.spark.network.util.ByteArrayWritableChannel
import org.apache.spark.util.Utils
import org.apache.spark.util.io.ChunkedByteBuffer

class ChunkedByteBufferSuite extends SparkFunSuite {
Expand Down Expand Up @@ -56,23 +56,6 @@ class ChunkedByteBufferSuite extends SparkFunSuite {
assert(chunkedByteBuffer.getChunks().head.position() === 0)
}

test("benchmark") {
val buffer100 = ByteBuffer.allocate(1024 * 1024 * 100)
val buffer30 = ByteBuffer.allocate(1024 * 1024 * 30)
val chunkedByteBuffer = new ChunkedByteBuffer(Array.fill(5)(buffer100))
var starTime = System.currentTimeMillis()
for (i <- 1 to 10) {
chunkedByteBuffer.writeFully(new ByteArrayWritableChannel(chunkedByteBuffer.size.toInt))
}
// scalastyle:off
System.out.println(System.currentTimeMillis() - starTime)
starTime = System.currentTimeMillis()
for (i <- 1 to 10) {
chunkedByteBuffer.writeWithSlice(new ByteArrayWritableChannel(chunkedByteBuffer.size.toInt))
}
System.out.println(System.currentTimeMillis() - starTime)
}

test("toArray()") {
val empty = ByteBuffer.wrap(Array.empty[Byte])
val bytes = ByteBuffer.wrap(Array.tabulate(8)(_.toByte))
Expand Down