Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
165eab1
[SPARK-3453] Refactor Netty module to use BlockTransferService.
rxin Sep 9, 2014
1760d32
Use Epoll.isAvailable in BlockServer as well.
rxin Sep 9, 2014
2b44cf1
Added more documentation.
rxin Sep 9, 2014
064747b
Reference count buffers and clean them up properly.
rxin Sep 10, 2014
b5c8d1f
Fixed ShuffleBlockFetcherIteratorSuite.
rxin Sep 10, 2014
108c9ed
Forgot to add TestSerializer to the commit list.
rxin Sep 10, 2014
1be4e8e
Shorten NioManagedBuffer and NettyManagedBuffer class names.
rxin Sep 10, 2014
cb589ec
Added more test cases covering cleanup when fault happens in ShuffleB…
rxin Sep 11, 2014
5cd33d7
Fixed style violation.
rxin Sep 11, 2014
9e0cb87
Fixed BlockClientHandlerSuite
rxin Sep 11, 2014
d23ed7b
Incorporated feedback from Norman:
rxin Sep 12, 2014
b2f3281
Added connection pooling.
rxin Sep 12, 2014
14323a5
Removed BlockManager.getLocalShuffleFromDisk.
rxin Sep 12, 2014
f0a16e9
Fixed test hanging.
rxin Sep 12, 2014
519d64d
Mark private package visibility and MimaExcludes.
rxin Sep 12, 2014
c066309
Implement java.io.Closeable interface.
rxin Sep 13, 2014
6afc435
Added logging.
rxin Sep 17, 2014
f63fb4c
Add more debug message.
rxin Sep 29, 2014
d68f328
Logging close() in case close() fails.
rxin Sep 29, 2014
1bdd7ee
Fixed tests.
rxin Sep 29, 2014
bec4ea2
Removed OIO and added num threads settings.
rxin Sep 29, 2014
4b18db2
Copy the buffer in fetchBlockSync.
rxin Sep 29, 2014
a0518c7
Implemented block uploads.
rxin Sep 30, 2014
407e59a
Fix style violation.
rxin Sep 30, 2014
f6c220d
Merge with latest master.
rxin Sep 30, 2014
5d98ce3
Flip buffer.
rxin Sep 30, 2014
f7e7568
Fixed spark.shuffle.io.receiveBuffer setting.
rxin Sep 30, 2014
29c6dcf
[SPARK-3453] Netty-based BlockTransferService, extracted from Spark core
aarondav Oct 6, 2014
ae4083a
[SPARK-2805] Upgrade Akka to 2.3.4
avati Oct 10, 2014
020691e
[SPARK-3886] [PySpark] use AutoBatchedSerializer by default
davies Oct 10, 2014
2c5d9dc
HOTFIX: Fix build issue with Akka 2.3.4 upgrade.
pwendell Oct 10, 2014
5b5dbe6
[SPARK-2924] Required by scala 2.11, only one fun/ctor amongst overri…
ScrapCodes Oct 11, 2014
8dc1ded
[SPARK-3867][PySpark] ./python/run-tests failed when it run with Pyth…
cocoatomo Oct 11, 2014
aa58f67
[SPARK-3909][PySpark][Doc] A corrupted format in Sphinx documents and…
cocoatomo Oct 11, 2014
939f276
Attempt to make comm. bidirectional
aarondav Oct 12, 2014
dd420fd
Merge branch 'master' of https://github.com/apache/spark into netty-test
aarondav Oct 17, 2014
7b7a26c
Fix Nio compile issue
aarondav Oct 17, 2014
d236dfd
Remove no-op serializer :)
aarondav Oct 17, 2014
9da0bc1
Add RPC unit tests
aarondav Oct 17, 2014
ccd4959
Don't throw exception if client immediately fails
aarondav Oct 17, 2014
e5675a4
Fail outstanding RPCs as well
aarondav Oct 18, 2014
322dfc1
Address Reynold's comments, including major rename
aarondav Oct 27, 2014
8dfcceb
Merge branch 'master' of https://github.com/apache/spark into netty
aarondav Oct 27, 2014
14e37f7
Address Reynold's comments
aarondav Oct 28, 2014
0c5bca2
Merge branch 'master' of https://github.com/apache/spark into netty
aarondav Oct 28, 2014
2b0d1c0
100ch
aarondav Oct 28, 2014
4a204b8
Fail block fetches if client connection fails
aarondav Oct 29, 2014
d7be11b
Turn netty on by default
aarondav Oct 29, 2014
cadfd28
Turn netty off by default
aarondav Oct 29, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added logging.
  • Loading branch information
rxin authored and aarondav committed Oct 10, 2014
commit 6afc435037a0448d6eb243bd18411ef25e3a2cf7
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import io.netty.channel.socket.nio.NioSocketChannel
import io.netty.channel.socket.oio.OioSocketChannel
import io.netty.util.internal.PlatformDependent

import org.apache.spark.SparkConf
import org.apache.spark.{Logging, SparkConf}
import org.apache.spark.util.Utils


Expand All @@ -42,7 +42,7 @@ import org.apache.spark.util.Utils
* for the same remote host. It also shares a single worker thread pool for all [[BlockClient]]s.
*/
private[netty]
class BlockClientFactory(val conf: NettyConfig) extends Closeable {
class BlockClientFactory(val conf: NettyConfig) extends Logging with Closeable {

def this(sparkConf: SparkConf) = this(new NettyConfig(sparkConf))

Expand Down Expand Up @@ -102,6 +102,8 @@ class BlockClientFactory(val conf: NettyConfig) extends Closeable {
return cachedClient
}

logInfo(s"Creating new connection to $remoteHost:$remotePort")

// There is a chance two threads are creating two different clients connecting to the same host.
// But that's probably ok ...

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,8 @@ class BlockServer(conf: NettyConfig, dataProvider: BlockDataManager)
_port = addr.getPort
// _hostName = addr.getHostName
_hostName = Utils.localHostName()

logInfo(s"Server started ${_hostName}:${_port}")
}

/** Shutdown the server. */
Expand Down