Skip to content

Unify GraphImpl RDDs + other graph load optimizations #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ankurdave wants to merge 12 commits into from
Closed

Unify GraphImpl RDDs + other graph load optimizations #137

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7402177
Handle empty partition iterators
ankurdave Apr 9, 2014
5a5c52a
SPARK-1329: Create pid2vid with correct number of partitions
ankurdave Apr 9, 2014
a142d14
Merge branch 'fix-pid2vid-size' into unify-rdds
ankurdave Apr 10, 2014
777654e
Merge branch 'handle-empty-partitions' into unify-rdds
ankurdave Apr 10, 2014
d419f00
Log current Pregel iteration
ankurdave Apr 20, 2014
3860a49
In Analytics, take PageRank numIter
ankurdave Apr 20, 2014
dc5512a
In GraphLoader, coalesce to minEdgePartitions
ankurdave Apr 20, 2014
74349e9
Unify GraphImpl RDDs + other graph load optimizations
ankurdave Apr 13, 2014
7c6ecfe
Show RDD name in stage table for easier debugging
ankurdave Apr 14, 2014
d6c8a9e
Custom serializer for RoutingTableMessage
ankurdave Apr 20, 2014
cccff33
Use *AggMsgSerializer for all RDD[(VertexId, VD)] shuffles
ankurdave Apr 20, 2014
f84585c
Faster EdgePartition building
ankurdave Apr 21, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Log current Pregel iteration 
(cherry picked from commit 54cbd2b)

Conflicts:
	graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
  • Loading branch information
@ankurdave
ankurdave committed Apr 21, 2014
commit d419f00727d16fad3a08d604cdb01a11bd563386
6 changes: 5 additions & 1 deletion graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.graphx

import scala.reflect.ClassTag
import org.apache.spark.Logging


/**
Expand Down Expand Up @@ -52,7 +53,7 @@ import scala.reflect.ClassTag
* }}}
*
*/
object Pregel {
object Pregel extends Logging {

/**
* Execute a Pregel-like iterative vertex-parallel abstraction. The
Expand Down Expand Up @@ -142,6 +143,9 @@ object Pregel {
// hides oldMessages (depended on by newVerts), newVerts (depended on by messages), and the
// vertices of prevG (depended on by newVerts, oldMessages, and the vertices of g).
activeMessages = messages.count()

logInfo("Pregel finished iteration " + i)

// Unpersist the RDDs hidden by newly-materialized RDDs
oldMessages.unpersist(blocking=false)
newVerts.unpersist(blocking=false)
Expand Down