Skip to content

[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up when unregisterShuffle. #31664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yikf wants to merge 11 commits into from
Closed

[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up when unregisterShuffle. #31664

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bff7afd
[SPARK-34395][SQL]Clean up unused code for code simplifications.
Feb 7, 2021
da47187
Merge remote-tracking branch 'origin/master'
Feb 26, 2021
da8facf
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Feb 26, 2021
1e9c3b6
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Feb 26, 2021
79f9a70
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Feb 26, 2021
69467ee
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Feb 27, 2021
7319255
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Mar 1, 2021
e53ee15
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Mar 2, 2021
a86b5ab
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up…
Mar 3, 2021
65be2c7
[SPARK-34541][CORE] for test.
Mar 4, 2021
cd907bb
[SPARK-34541][CORE] revert
Mar 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up… 
… when unregisterShuffle.
  • Loading branch information
yikf committed Feb 27, 2021
commit 69467ee6c70d1074b3a79d093cde365e5ed648a3
4 changes: 2 additions & 2 deletions core/src/test/scala/org/apache/spark/shuffle/sort/SortShuffleManagerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import org.apache.spark.serializer.{JavaSerializer, KryoSerializer, Serializer}
* Tests for the fallback logic in UnsafeShuffleManager. Actual tests of shuffling data are
* performed in other suites.
*/
class SortShuffleManagerSuite extends SparkFunSuite with Matchers {
class SortShuffleManagerSuite extends SparkFunSuite with Matchers with LocalSparkContext {

private def doReturn(value: Any) = org.mockito.Mockito.doReturn(value, Seq.empty: _*)

Expand Down Expand Up @@ -143,7 +143,7 @@ class SortShuffleManagerSuite extends SparkFunSuite with Matchers {
withTempDir { tmpDir =>
val conf = new SparkConf(loadDefaults = false)
conf.set("spark.local.dir", tmpDir.getAbsolutePath)
val sc: SparkContext = new SparkContext("local", "SPARK-34541", conf)
sc = new SparkContext("local", "SPARK-34541", conf)
val rdd = sc.parallelize(1 to 10, 1).map(x => (x, x))
// Create a shuffleRdd
val shuffledRdd = new ShuffledRDD[Int, Int, Int](rdd, new HashPartitioner(4))
Expand Down