[SPARK-25954][SS][FOLLOWUP][test-maven] Add Zookeeper 3.4.7 test dependency to Kafka modules #23119

dongjoon-hyun · 2018-11-22T19:30:04Z

What changes were proposed in this pull request?

This is a followup of #23099 . After upgrading to Kafka 2.1.0, maven test fails due to Zookeeper test dependency while sbt test succeeds.

The root cause is that the embedded Kafka server is using Zookeepr 3.4.7 API while Apache Spark provides Zookeeper 3.4.6. This PR adds a test dependency.

KafkaMicroBatchV2SourceSuite:
*** RUN ABORTED ***
...
org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123)
  ...
  Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
  at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
  at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
  at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
  at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120)
  at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116)
  ...

How was this patch tested?

Pass the maven Jenkins test.

…fka-0-10 module https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html KafkaMicroBatchV2SourceSuite: *** RUN ABORTED *** java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) at kafka.server.KafkaServer.startup(KafkaServer.scala:202) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116) at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2221) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:155) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2213) at org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123) ... Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) at kafka.server.KafkaServer.startup(KafkaServer.scala:202) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116) ... a

gaborgsomogyi · 2018-11-22T19:47:07Z

Wondering why tests passed with sbt. Does sbt handles deps in a different way?

dongjoon-hyun · 2018-11-22T20:12:19Z

sbt and maven have different behavior to resolve the dependencies. So, in Apache Spark community, maven is the official one although we support both and sbt is used in PR builder.

dongjoon-hyun · 2018-11-22T20:13:26Z

Retest this please.

dongjoon-hyun · 2018-11-22T20:14:35Z

I forgot [test-maven] is a case-sensitive tag. The previous one went to sbt Jenkins. Hopefully, next one will go to 'maven' Jenkins.

SparkQA · 2018-11-22T20:27:13Z

Test build #99198 has finished for PR 23119 at commit 9c5d796.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-22T21:18:28Z

Retest this please.

SparkQA · 2018-11-22T21:32:29Z

Test build #99199 has finished for PR 23119 at commit 9c5d796.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-22T21:36:10Z

ChunkFetchIntegrationSuite seems to be very flacky in Maven build. This causes two consequtive irrelevant failures in this PR.

[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.183 s <<< FAILURE! - in org.apache.spark.network.ChunkFetchIntegrationSuite
[ERROR] fetchBothChunks(org.apache.spark.network.ChunkFetchIntegrationSuite)  Time elapsed: 5.069 s  <<< FAILURE!
java.lang.AssertionError: Timeout getting response from the server
	at org.apache.spark.network.ChunkFetchIntegrationSuite.fetchChunks(ChunkFetchIntegrationSuite.java:176)
	at org.apache.spark.network.ChunkFetchIntegrationSuite.fetchBothChunks(ChunkFetchIntegrationSuite.java:210)

dongjoon-hyun · 2018-11-22T21:37:03Z

Retest this please.

gaborgsomogyi · 2018-11-22T21:38:26Z

I've seen similar flakyness with sbt as well.

SparkQA · 2018-11-22T21:52:19Z

Test build #99200 has finished for PR 23119 at commit 9c5d796.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-22T22:51:04Z

Retest this please.

SparkQA · 2018-11-22T23:38:06Z

Test build #99196 has finished for PR 23119 at commit 9c5d796.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-22T23:48:29Z

The previous successful one was sbt.
The current on-going one fails with another flacky test, SPARK-25903.

BarrierTaskContextSuite:
...
- throw exception if the number of barrier() calls are not the same on every task *** FAILED ***

dongjoon-hyun · 2018-11-22T23:48:36Z

Retest this please.

dongjoon-hyun · 2018-11-23T02:09:28Z

The current one hangs on BroadcastSuite.

dongjoon-hyun · 2018-11-23T02:09:34Z

Retest this please.

SparkQA · 2018-11-23T02:23:17Z

Test build #99204 has finished for PR 23119 at commit 9c5d796.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-23T02:30:19Z

Retest this please

SparkQA · 2018-11-23T03:18:19Z

Test build #99203 has finished for PR 23119 at commit 9c5d796.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-23T05:35:37Z

Test build #99202 has finished for PR 23119 at commit 9c5d796.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-23T06:12:04Z

Hi, @cloud-fan . Could you review this, please?
Test build #99205 has started is almost passed. It's at SparkR testing stage.

I'd like to merge this because this causes UT failures in both maven builds (master-maven and master-maven-scala-2.12). Also, this PR suffers from flaky tests a lot today.

SparkQA · 2018-11-23T06:34:39Z

Test build #99205 has finished for PR 23119 at commit 9c5d796.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-23T06:44:30Z

Merged to master to recover master branch Maven testings.

gaborgsomogyi

A bit late but LGTM.

HyukjinKwon · 2018-11-23T10:52:05Z

LGTM too

dongjoon-hyun · 2018-11-23T17:44:20Z

Thanks, @gaborgsomogyi and @HyukjinKwon .

…ndency to Kafka modules ## What changes were proposed in this pull request? This is a followup of apache#23099 . After upgrading to Kafka 2.1.0, maven test fails due to Zookeeper test dependency while sbt test succeeds. - [sbt test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/5203/) - [maven test on master branch](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5653/) The root cause is that the embedded Kafka server is using [Zookeepr 3.4.7 API](https://zookeeper.apache.org/doc/r3.4.7/api/org/apache/zookeeper/AsyncCallback.MultiCallback.html ) while Apache Spark provides Zookeeper 3.4.6. This PR adds a test dependency. ``` KafkaMicroBatchV2SourceSuite: *** RUN ABORTED *** ... org.apache.spark.sql.kafka010.KafkaTestUtils.setupEmbeddedKafkaServer(KafkaTestUtils.scala:123) ... Cause: java.lang.ClassNotFoundException: org.apache.zookeeper.AsyncCallback$MultiCallback at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) at kafka.server.KafkaServer.startup(KafkaServer.scala:202) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2(KafkaTestUtils.scala:120) at org.apache.spark.sql.kafka010.KafkaTestUtils.$anonfun$setupEmbeddedKafkaServer$2$adapted(KafkaTestUtils.scala:116) ... ``` ## How was this patch tested? Pass the maven Jenkins test. Closes apache#23119 from dongjoon-hyun/SPARK-25954-2. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun changed the title ~~[SPARK-25954][SS][FOLLOWUP][TEST-MAVEN] Add Zookeeper 3.4.7 test dependency to Kafka modules~~ [SPARK-25954][SS][FOLLOWUP][test-maven] Add Zookeeper 3.4.7 test dependency to Kafka modules Nov 22, 2018

asfgit closed this in 1d3dd58 Nov 23, 2018

dongjoon-hyun deleted the SPARK-25954-2 branch November 23, 2018 07:19

gaborgsomogyi reviewed Nov 23, 2018

View reviewed changes

[SPARK-25954][SS][FOLLOWUP][test-maven] Add Zookeeper 3.4.7 test dependency to Kafka modules #23119

[SPARK-25954][SS][FOLLOWUP][test-maven] Add Zookeeper 3.4.7 test dependency to Kafka modules #23119

Uh oh!

Conversation

dongjoon-hyun commented Nov 22, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gaborgsomogyi commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

SparkQA commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

SparkQA commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

gaborgsomogyi commented Nov 22, 2018

Uh oh!

SparkQA commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

SparkQA commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018

Uh oh!

SparkQA commented Nov 23, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018

Uh oh!

SparkQA commented Nov 23, 2018

Uh oh!

SparkQA commented Nov 23, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 23, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018

Uh oh!

gaborgsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Nov 23, 2018

Uh oh!

dongjoon-hyun commented Nov 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Nov 23, 2018 •

edited

Loading