Skip to content

[SPARK-5259][CORE]Make sure shuffle metadata already in mapOutPutTracker while submitting tasks of the shuffleMapStage #4055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
suyanNone wants to merge 17 commits into from
Closed

[SPARK-5259][CORE]Make sure shuffle metadata already in mapOutPutTracker while submitting tasks of the shuffleMapStage #4055

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
641936f
Make sure mapStage.pendingtasks is set() while MapStage.isAvailable i…
suyanNone Jan 15, 2015
741ab4d
Add sub-class canEqual to make ShuffleMapTask not equal Result Task w…
suyanNone Jan 26, 2015
5ec8a82
Refine
suyanNone Jan 29, 2015
9044720
add parameter type
suyanNone Jan 30, 2015
03db624
Refine
suyanNone Jan 30, 2015
8dfdc18
Refine log message use string interpolation
suyanNone Feb 27, 2015
c286f7a
Refine with the latest spark
suyanNone Feb 27, 2015
dcf1533
Refine solution and fix bug in code
suyanNone Jun 18, 2015
27da8e7
Refine codeStyle
suyanNone Jun 18, 2015
bd5fec4
Refine Tests
suyanNone Jun 19, 2015
4dbe4d3
Refine assert exception
suyanNone Jun 19, 2015
9dfff63
Refine the test 'ignore late map task completions'
suyanNone Jun 19, 2015
e1e0b66
Refine the testcase
suyanNone Jul 3, 2015
2379250
Refine scala style
suyanNone Jul 3, 2015
314873a
Only tracker partitionId to instead of Task
suyanNone Jul 21, 2015
7ae128e
Refine test name
suyanNone Jul 21, 2015
b2df3fd
refine suite code
suyanNone Jul 21, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refine assert exception
  • Loading branch information
@suyanNone
suyanNone committed Jul 21, 2015
commit 4dbe4d32c9364d8c11c417e0e91688ee20160dab
10 changes: 7 additions & 3 deletions core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

package org.apache.spark.scheduler

import org.apache.spark.shuffle.MetadataFetchFailedException

Copy link
Contributor

@squito squito Jul 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not used, delete

import scala.collection.mutable.{ArrayBuffer, HashSet, HashMap, Map}
import scala.language.reflectiveCalls
import scala.util.control.NonFatal
Expand Down Expand Up @@ -776,9 +778,11 @@ class DAGSchedulerSuite
runEvent(CompletionEvent(taskSets(0).tasks(2), Success,
makeMapStatus("hostC", shuffleMapRdd.partitions.size), null, createFakeTaskInfo(), null))

val thrown = intercept[Exception] { mapOutputTracker.getServerStatuses(0, 2) }
// Assert not throw MetadataFetchFailedException: Missing an output location for shuffle 0
assert(thrown == null)
try {
mapOutputTracker.getServerStatuses(0, 2)
} catch {
case e: MetadataFetchFailedException => fail("Should not throw metadataFetchFailedException")
}

runEvent(CompletionEvent(taskSets(1).tasks(0), Success,
makeMapStatus("hostC", reduceRdd.partitions.size), null, createFakeTaskInfo(), null))
Expand Down