Skip to content

[SPARK-6521][Core] Bypass unnecessary network access if block managers share an identical host #9478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
maropu wants to merge 6 commits into from
Closed

[SPARK-6521][Core] Bypass unnecessary network access if block managers share an identical host #9478

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
64f8896
Bypass unnecessary network access if block managers share an identica…
maropu Nov 4, 2015
a93ec36
Disable the bypassing optimization if external shuffle service enabled
maropu Nov 5, 2015
bee23aa
Fix bugs in tests
maropu Nov 5, 2015
b1126aa
Apply comments
maropu Nov 6, 2015
ba94687
Apply style comments
maropu Nov 26, 2015
303abcd
Fix the interface issues
maropu Nov 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply comments
  • Loading branch information
@maropu
maropu committed Nov 27, 2015
commit b1126aab64d810a8f165f3f2f46b22ec7bb597ba
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,8 +207,8 @@ private[spark] class IndexShuffleBlockResolver(
}

override def getBlockData(
blockId: ShuffleBlockId, blockManagerId: BlockManagerId)
: ManagedBuffer = {
blockId: ShuffleBlockId,
blockManagerId: BlockManagerId): ManagedBuffer = {
// The block is actually going to be a range of a single map output file for this map, so
// find out the consolidated file, then the offset within that from our index
val indexFile = getIndexFile(blockId.shuffleId, blockId.mapId, blockManagerId)
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/storage/BlockManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ import org.apache.spark.network.netty.SparkTransportConf
import org.apache.spark.network.shuffle.ExternalShuffleClient
import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo
import org.apache.spark.rpc.RpcEnv
import org.apache.spark.serializer.{Serializer, SerializerInstance}
import org.apache.spark.serializer.{SerializerInstance, Serializer}
import org.apache.spark.shuffle.ShuffleManager
import org.apache.spark.util._

Expand Down
12 changes: 6 additions & 6 deletions core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@

package org.apache.spark.storage

import org.apache.spark.rpc.RpcEndpointRef
import org.apache.spark.storage.BlockManagerMessages._
import org.apache.spark.util.{RpcUtils, ThreadUtils}
import org.apache.spark.{Logging, SparkConf, SparkException}

import scala.collection.Iterable
import scala.collection.generic.CanBuildFrom
import scala.concurrent.Future
import scala.concurrent.{Await, Future}

import org.apache.spark.rpc.RpcEndpointRef
import org.apache.spark.{Logging, SparkConf, SparkException}
import org.apache.spark.storage.BlockManagerMessages._
import org.apache.spark.util.{ThreadUtils, RpcUtils}

private[spark]
class BlockManagerMaster(
Expand Down