Skip to content

[SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors" #46989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bozhang2820 wants to merge 7 commits into from
Closed

[SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors" #46989

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8facb37
Fix test
bozhang2820 Jun 14, 2024
a5e0623
Actual fix
bozhang2820 Jun 18, 2024
c8ddecd
Add verify
bozhang2820 Jun 19, 2024
741a610
trigger CI
bozhang2820 Jun 19, 2024
e51d480
again
bozhang2820 Jun 20, 2024
54b73f7
and again
bozhang2820 Jun 20, 2024
d3b739b
again
bozhang2820 Jun 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,9 +392,11 @@ class ShuffleBlockFetcherIteratorSuite extends SparkFunSuite with PrivateMethodT

configureMockTransfer(Map())
val iterator = createShuffleBlockIteratorWithDefaults(
Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1))
Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1)),
blockManager = Some(blockManager)
Copy link
Member

@Ngone51 Ngone51 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockManager is actually the default value of createShuffleBlockIteratorWithDefaults(). So it looks like there is no actual difference made by this change.

Copy link
Contributor Author

@bozhang2820 bozhang2820 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do change the return of blockManager.hostLocalDirManager to Some(hostLocalDirManager), so that the blocks are treated as host-local.

Copy link
Member

@Ngone51 Ngone51 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. That makes sense.

Copy link
Contributor Author

@bozhang2820 bozhang2820 Jun 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally that the correct code path was reached in partitionBlocksByFetchMode. I can add an additional check to verify that mockExternalBlockStoreClient.getHostLocalDirs was called in the test.

)
intercept[FetchFailedException] { iterator.next() }
verify(mockExternalBlockStoreClient, times(1)).getHostLocalDirs(any(), any(), any(), any())
}

test("Hit maxBytesInFlight limitation before maxBlocksInFlightPerAddress") {
Expand Down