Skip to content

[SPARK-3453] Netty-based BlockTransferService #2330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rxin wants to merge 34 commits into from
Closed

[SPARK-3453] Netty-based BlockTransferService #2330

Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5bb88f5
[SPARK-3453] Refactor Netty module to use BlockTransferService.
rxin Sep 9, 2014
9b3b397
Use Epoll.isAvailable in BlockServer as well.
rxin Sep 9, 2014
dd783ff
Added more documentation.
rxin Sep 9, 2014
b5b380e
Reference count buffers and clean them up properly.
rxin Sep 10, 2014
1474824
Fixed ShuffleBlockFetcherIteratorSuite.
rxin Sep 10, 2014
b404da3
Forgot to add TestSerializer to the commit list.
rxin Sep 10, 2014
fbf882d
Shorten NioManagedBuffer and NettyManagedBuffer class names.
rxin Sep 10, 2014
b32c3fe
Added more test cases covering cleanup when fault happens in ShuffleB…
rxin Sep 11, 2014
d135fa3
Fixed style violation.
rxin Sep 11, 2014
1e0d277
Fixed BlockClientHandlerSuite
rxin Sep 11, 2014
6e84cb2
Merge branch 'master' into netty-blockTransferService
rxin Sep 11, 2014
55266d1
Incorporated feedback from Norman:
rxin Sep 12, 2014
f83611e
Added connection pooling.
rxin Sep 12, 2014
6ddaa5d
Removed BlockManager.getLocalShuffleFromDisk.
rxin Sep 12, 2014
8295561
Fixed test hanging.
rxin Sep 12, 2014
d7d0aac
Mark private package visibility and MimaExcludes.
rxin Sep 12, 2014
29fe0cc
Implement java.io.Closeable interface.
rxin Sep 13, 2014
e92dad7
Merge branch 'master' into netty-blockTransferService
rxin Sep 17, 2014
a79a259
Added logging.
rxin Sep 17, 2014
088ed8a
Fixed error message.
rxin Sep 17, 2014
323dfec
Add more debug message.
rxin Sep 29, 2014
5814292
Logging close() in case close() fails.
rxin Sep 29, 2014
f23e682
Merge branch 'master' into netty-blockTransferService
rxin Sep 29, 2014
ba8c441
Fixed tests.
rxin Sep 29, 2014
ca88068
Merge branch 'buffer-debug' into netty-blockTransferService
rxin Sep 29, 2014
dfc2c34
Removed OIO and added num threads settings.
rxin Sep 29, 2014
3fbfd3f
Merge branch 'master' into netty-blockTransferService
rxin Sep 29, 2014
69f5d0a
Copy the buffer in fetchBlockSync.
rxin Sep 29, 2014
bc9ed22
Implemented block uploads.
rxin Sep 30, 2014
a3a09f6
Fix style violation.
rxin Sep 30, 2014
0140d6e
Merge branch 'master' into netty-blockTransferService
rxin Sep 30, 2014
0dae310
Merge with latest master.
rxin Sep 30, 2014
ad09236
Flip buffer.
rxin Sep 30, 2014
bdab2c7
Fixed spark.shuffle.io.receiveBuffer setting.
rxin Sep 30, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixed style violation.
  • Loading branch information
@rxin
rxin committed Sep 11, 2014
commit d135fa38801467b0dd870063c00103ddd45438c7
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/network/netty/BlockServer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ class BlockServer(conf: NettyConfig, dataProvider: BlockDataManager) extends Log

val addr = channelFuture.channel.localAddress.asInstanceOf[InetSocketAddress]
_port = addr.getPort
//_hostName = addr.getHostName
// _hostName = addr.getHostName
_hostName = Utils.localHostName()
}

Expand Down