Skip to content

Unify GraphImpl RDDs + other graph load optimizations #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ankurdave wants to merge 17 commits into from
Closed

Unify GraphImpl RDDs + other graph load optimizations #497

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d64e8d4
Log current Pregel iteration
ankurdave Apr 20, 2014
62c7b78
In Analytics, take PageRank numIter
ankurdave Apr 20, 2014
d6d60e2
In GraphLoader, coalesce to minEdgePartitions
ankurdave Apr 20, 2014
e4fbd32
Unify GraphImpl RDDs + other graph load optimizations
ankurdave Apr 13, 2014
ae36110
Fix style errors
ankurdave Apr 23, 2014
10b3596
Clean up public API
ankurdave Apr 23, 2014
2a928b2
Set locality wait
ankurdave Apr 25, 2014
0d3584c
EdgePartition.size should be val
ankurdave Apr 25, 2014
c88b269
Revert upgradeIterator to if-in-a-loop
ankurdave May 7, 2014
04d3ae5
Convert implicit parameter to context bound
ankurdave May 7, 2014
75af062
Add explicit return types
ankurdave May 7, 2014
57202e8
Replace case with pair parameter
ankurdave May 7, 2014
a04765c
Remove unnecessary toOps call
ankurdave May 7, 2014
13ac845
Merge remote-tracking branch 'apache-spark/master' into unify-rdds
ankurdave May 7, 2014
5ba8789
Add GraphX upgrade guide from Spark 0.9.1
ankurdave May 7, 2014
4933e2e
Exclude RoutingTable from binary compatibility check
ankurdave May 7, 2014
332ab43
Merge remote-tracking branch 'apache-spark/master' into unify-rdds
ankurdave May 10, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Merge remote-tracking branch 'apache-spark/master' into unify-rdds 
Also update RoutingTableMessageSerializer to pass ClassTags.
  • Loading branch information
@ankurdave
ankurdave committed May 10, 2014
commit 332ab43f388a1e0a1e445971289df80c68072ad6
4 changes: 2 additions & 2 deletions graphx/src/main/scala/org/apache/spark/graphx/impl/Serializers.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ class RoutingTableMessageSerializer extends Serializer with Serializable {

override def serializeStream(s: OutputStream): SerializationStream =
new ShuffleSerializationStream(s) {
def writeObject[T](t: T): SerializationStream = {
def writeObject[T: ClassTag](t: T): SerializationStream = {
val msg = t.asInstanceOf[RoutingTableMessage]
writeVarLong(msg.vid, optimizePositive = false)
writeUnsignedVarInt(msg.pid)
Expand All @@ -45,7 +45,7 @@ class RoutingTableMessageSerializer extends Serializer with Serializable {

override def deserializeStream(s: InputStream): DeserializationStream =
new ShuffleDeserializationStream(s) {
override def readObject[T](): T = {
override def readObject[T: ClassTag](): T = {
val a = readVarLong(optimizePositive = false)
val b = readUnsignedVarInt()
val c = s.read()
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.