Skip to content

[SPARK-25214][SS]Fix the issue that Kafka v2 source may return duplicated records when failOnDataLoss=false #22207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zsxwing wants to merge 3 commits into from
Closed

[SPARK-25214][SS]Fix the issue that Kafka v2 source may return duplicated records when failOnDataLoss=false #22207

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f2d4d67
Fix the issue that Kafka v2 source may return duplicated records when…
zsxwing Aug 23, 2018
e968159
style
zsxwing Aug 23, 2018
3515275
address
zsxwing Aug 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
style
  • Loading branch information
@zsxwing
zsxwing committed Aug 23, 2018
commit e968159a1fcedadd6885a424656e1c8c5afb1aa7
2 changes: 1 addition & 1 deletion ...-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ class KafkaDontFailOnDataLossSuite extends KafkaMissingOffsetsTest {
* @param writeToTable the function to write the specified [[DataFrame]] to the given table.
*/
private def verifyMissingOffsetsDontCauseDuplicatedRecords(
testStreamingQuery: Boolean)(writeToTable: (DataFrame, String) => Unit): Unit = {
testStreamingQuery: Boolean)(writeToTable: (DataFrame, String) => Unit): Unit = {
val topic = newTopic()
testUtils.createTopic(topic, partitions = 1)
testUtils.sendMessages(topic, (0 until 50).map(_.toString).toArray)
Expand Down