Skip to content

Unify GraphImpl RDDs + other graph load optimizations #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ankurdave wants to merge 12 commits into from
Closed

Unify GraphImpl RDDs + other graph load optimizations #137

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7402177
Handle empty partition iterators
ankurdave Apr 9, 2014
5a5c52a
SPARK-1329: Create pid2vid with correct number of partitions
ankurdave Apr 9, 2014
a142d14
Merge branch 'fix-pid2vid-size' into unify-rdds
ankurdave Apr 10, 2014
777654e
Merge branch 'handle-empty-partitions' into unify-rdds
ankurdave Apr 10, 2014
d419f00
Log current Pregel iteration
ankurdave Apr 20, 2014
3860a49
In Analytics, take PageRank numIter
ankurdave Apr 20, 2014
dc5512a
In GraphLoader, coalesce to minEdgePartitions
ankurdave Apr 20, 2014
74349e9
Unify GraphImpl RDDs + other graph load optimizations
ankurdave Apr 13, 2014
7c6ecfe
Show RDD name in stage table for easier debugging
ankurdave Apr 14, 2014
d6c8a9e
Custom serializer for RoutingTableMessage
ankurdave Apr 20, 2014
cccff33
Use *AggMsgSerializer for all RDD[(VertexId, VD)] shuffles
ankurdave Apr 20, 2014
f84585c
Faster EdgePartition building
ankurdave Apr 21, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Faster EdgePartition building
  • Loading branch information
@ankurdave
ankurdave committed Apr 21, 2014
commit f84585c3f8a081c53a46345ac3aeb7aa464fb524
9 changes: 6 additions & 3 deletions graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartitionBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import scala.util.Sorting

import org.apache.spark.graphx._
import org.apache.spark.graphx.util.collection.PrimitiveKeyOpenHashMap
import org.apache.spark.util.collection.PrimitiveVector
import org.apache.spark.util.collection.{BitSet, OpenHashSet, PrimitiveVector}

private[graphx]
class EdgePartitionBuilder[@specialized(Long, Int, Double) ED: ClassTag, VD: ClassTag](
Expand Down Expand Up @@ -60,8 +60,11 @@ class EdgePartitionBuilder[@specialized(Long, Int, Double) ED: ClassTag, VD: Cla
}
}

val vertexIds = srcIds.iterator ++ dstIds.iterator
val vertexPartition = VertexPartition(vertexIds.map(vid => (vid, defaultVertexAttr)))
val vidsIter = srcIds.iterator ++ dstIds.iterator
val vertexIds = new OpenHashSet[VertexId]
vidsIter.foreach(vid => vertexIds.add(vid))
val vertexPartition = new VertexPartition(
vertexIds, new Array[VD](vertexIds.capacity), new BitSet(vertexIds.capacity))
new EdgePartition(srcIds, dstIds, data, index, vertexPartition)
}
}