Skip to content

Unify GraphImpl RDDs + other graph load optimizations #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ankurdave wants to merge 17 commits into from
Closed

Unify GraphImpl RDDs + other graph load optimizations #497

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d64e8d4
Log current Pregel iteration
ankurdave Apr 20, 2014
62c7b78
In Analytics, take PageRank numIter
ankurdave Apr 20, 2014
d6d60e2
In GraphLoader, coalesce to minEdgePartitions
ankurdave Apr 20, 2014
e4fbd32
Unify GraphImpl RDDs + other graph load optimizations
ankurdave Apr 13, 2014
ae36110
Fix style errors
ankurdave Apr 23, 2014
10b3596
Clean up public API
ankurdave Apr 23, 2014
2a928b2
Set locality wait
ankurdave Apr 25, 2014
0d3584c
EdgePartition.size should be val
ankurdave Apr 25, 2014
c88b269
Revert upgradeIterator to if-in-a-loop
ankurdave May 7, 2014
04d3ae5
Convert implicit parameter to context bound
ankurdave May 7, 2014
75af062
Add explicit return types
ankurdave May 7, 2014
57202e8
Replace case with pair parameter
ankurdave May 7, 2014
a04765c
Remove unnecessary toOps call
ankurdave May 7, 2014
13ac845
Merge remote-tracking branch 'apache-spark/master' into unify-rdds
ankurdave May 7, 2014
5ba8789
Add GraphX upgrade guide from Spark 0.9.1
ankurdave May 7, 2014
4933e2e
Exclude RoutingTable from binary compatibility check
ankurdave May 7, 2014
332ab43
Merge remote-tracking branch 'apache-spark/master' into unify-rdds
ankurdave May 10, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Log current Pregel iteration
  • Loading branch information
@ankurdave
ankurdave committed Apr 23, 2014
commit d64e8d49ec43d2fd6bde519b8aeafb5cc4f1be61
6 changes: 5 additions & 1 deletion graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark.graphx

import scala.reflect.ClassTag
import org.apache.spark.Logging


/**
Expand Down Expand Up @@ -52,7 +53,7 @@ import scala.reflect.ClassTag
* }}}
*
*/
object Pregel {
object Pregel extends Logging {

/**
* Execute a Pregel-like iterative vertex-parallel abstraction. The
Expand Down Expand Up @@ -142,6 +143,9 @@ object Pregel {
// hides oldMessages (depended on by newVerts), newVerts (depended on by messages), and the
// vertices of prevG (depended on by newVerts, oldMessages, and the vertices of g).
activeMessages = messages.count()

logInfo("Pregel finished iteration " + i)

// Unpersist the RDDs hidden by newly-materialized RDDs
oldMessages.unpersist(blocking=false)
newVerts.unpersist(blocking=false)
Expand Down