Skip to content

[SPARK-23747][Structured Streaming] Add EpochCoordinator unit tests #20983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
spaced4ndy wants to merge 5 commits into from
Closed

[SPARK-23747][Structured Streaming] Add EpochCoordinator unit tests #20983

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b5cd066
[SPARK-23747][Structured Streaming] Added EpochCoordinator unit tests
Apr 5, 2018
284795b
Passing raw numbers for epochs as parameters
Apr 10, 2018
7d85cf2
Added tests
Apr 10, 2018
89a1e11
Renamed method
Apr 11, 2018
8fa609c
Replaced SharedSparkSession with LocalSparkSession
Apr 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Renamed method
  • Loading branch information
Efim Poberezkin committed Apr 11, 2018
commit 89a1e11af0712c7557b45aad7bc2ffde95b529fc
6 changes: 3 additions & 3 deletions ...core/src/test/scala/org/apache/spark/sql/streaming/continuous/EpochCoordinatorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ class EpochCoordinatorSuite

makeSynchronousCall()

verifyCommitHasntHappened(1)
verifyNoCommitFor(1)
}

test("single epoch, all but one reader partition has reported an offset") {
Expand All @@ -99,7 +99,7 @@ class EpochCoordinatorSuite

makeSynchronousCall()

verifyCommitHasntHappened(1)
verifyNoCommitFor(1)
}
Copy link
Contributor

@jose-torres jose-torres Apr 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest adding test cases where all but one writer partition has committed, or all but one reader partition has reported an offset. In those cases we should verify that the StreamWriter and query commits haven't happened.


test("consequent epochs, messages for epoch (k + 1) arrive after messages for epoch k") {
Expand Down Expand Up @@ -214,7 +214,7 @@ class EpochCoordinatorSuite
orderVerifier.verify(query).commit(epoch)
}

private def verifyCommitHasntHappened(epoch: Long): Unit = {
private def verifyNoCommitFor(epoch: Long): Unit = {
verify(writer, never()).commit(eqTo(epoch), any())
verify(query, never()).commit(epoch)
}
Expand Down