[SPARK-9574][Streaming]Remove unnecessary contents of spark-streaming-XXX-assembly jars #8069

zsxwing · 2015-08-10T16:56:54Z

Removed contents already included in Spark assembly jar from spark-streaming-XXX-assembly jars.

…ing-flume-assembly

…ing-kafka-assembly

…ing-kinesis-asl-assembly

srowen · 2015-08-10T17:03:17Z

CC @harishreedharan

SparkQA · 2015-08-10T19:21:07Z

Test build #40296 has finished for PR 8069 at commit 9e06559.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PoissonSampler[T: ClassTag](

tdas · 2015-08-10T20:27:11Z

Jenkins, test this please.

SparkQA · 2015-08-10T23:07:32Z

Test build #40316 has finished for PR 8069 at commit 9e06559.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-08-10T23:54:30Z

@zsxwing Could you update this PR with MQTT fixes as well.

…ing-mqtt-assembly

tdas · 2015-08-11T05:39:22Z

pom.xml

Why this change?

This link says that common-lang was moved from common-lang group to org.apache.commons group.
http://mvnrepository.com/artifact/commons-lang/commons-lang

This is incorrect. The correct groupId for commans-lang 2 is commons-lang. However, no place used this one previously so it didn't break anything.

This link says that common-lang was moved from common-lang group to org.apache.commons group.
http://mvnrepository.com/artifact/commons-lang/commons-lang

Right. But it's under commons-lang3: http://mvnrepository.com/artifact/org.apache.commons/commons-lang3 since they cannot change the old artifact.

There are some libraries still using commons-lang 2.x.

SparkQA · 2015-08-11T06:46:05Z

Test build #40388 has finished for PR 8069 at commit 546450a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

zsxwing · 2015-08-11T07:19:30Z

retest this please

SparkQA · 2015-08-11T09:04:39Z

Test build #1441 has finished for PR 8069 at commit 546450a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-11T09:30:51Z

Test build #40425 has finished for PR 8069 at commit 546450a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-08-11T09:34:13Z

retest this please

zsxwing · 2015-08-11T10:34:27Z

retest this please

SparkQA · 2015-08-11T12:00:44Z

Test build #1449 has finished for PR 8069 at commit 546450a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-08-11T12:59:32Z

retest this please

zsxwing · 2015-08-11T13:31:03Z

retest this please

SparkQA · 2015-08-11T13:38:31Z

Test build #40445 timed out for PR 8069 at commit 546450a after a configured wait of 175m.

zsxwing · 2015-08-11T16:56:52Z

retest this please

SparkQA · 2015-08-11T19:59:18Z

Test build #40470 timed out for PR 8069 at commit 546450a after a configured wait of 175m.

zsxwing · 2015-08-12T01:01:49Z

retest this please.

SparkQA · 2015-08-18T10:10:32Z

Test build #1648 has finished for PR 8069 at commit 546450a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-08-18T10:45:15Z

retest this please

SparkQA · 2015-08-18T13:48:12Z

Test build #41123 has finished for PR 8069 at commit 546450a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-08-18T20:36:14Z

I am merging this to master and 1.5, thanks!

…ng-XXX-assembly jars Removed contents already included in Spark assembly jar from spark-streaming-XXX-assembly jars. Author: zsxwing <[email protected]> Closes #8069 from zsxwing/SPARK-9574. (cherry picked from commit bf1d661) Signed-off-by: Tathagata Das <[email protected]>

shivaram · 2015-08-19T07:15:57Z

@srowen @tdas It looks like this PR somehow changed the dependency tree and affected the documentation build that happens as a cron job on Jenkins.

The error log at https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/690/consoleFull is a run that happened after this PR and https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/689/consoleFull is a run from before

It looks like this PR somehow introduces a dependency on netty 3.2.2 through mqtt, curator etc.

Note: Unresolved dependencies path:�
   org.jboss.netty:netty:3.2.2.Final�
   +-org.apache.zookeeper:zookeeper:3.4.5 ((com.typesafe.sbt.pom.MavenHelper) 
+- org.apache.curator:curator-client:2.4.0�
+- org.apache.curator:curator-framework:2.4.0�
+- org.apache.curator:curator-recipes:2.4.0 ((com.typesafe.sbt.pom.MavenHelper) 
+- org.apache.spark:spark-core_2.10:1.5.0-SNAPSHOT�
+- org.apache.spark:spark-streaming_2.10:1.5.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper) 
+- org.apache.spark:spark-streaming-mqtt_2.10:1.5.0-SNAPSHOT 
+- org.apache.spark:spark-streaming-mqtt-assembly_2.10:1.5.0-SNAPSHOT�

Did some exclude rules change or something like that ?

cc @rxin

tdas · 2015-08-19T07:43:58Z

The issues is actually SBT failing to resolve the artifact in the m2 cache
even though the artifact is present. Basically the ivy2 cache of jboss
needs to blown away.

I am slowly fixing the issue on the jenkins workers. Since I am not able to
access the Jenkins workers directly (ssh access not working for me), I am
leveraging the Spark SBT master build to blow away cached jboss. And
whatever worker it is running, that worker is getting fixed. Here is what
fixes it.

rm -rf /home/jenkins/.ivy2/cache/org.jboss*
rm -rf /home/jenkins/.ivy2/cache/junit*

I have it added in the Spark-SBT-master -
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/configure
If you can login into the workers, would you mind running these two lines
on each of them?

On Wed, Aug 19, 2015 at 12:16 AM, Shivaram Venkataraman <
[email protected]> wrote:

@srowen https://github.com/srowen @tdas https://github.com/tdas It
looks like this PR somehow changed the dependency tree and affected the
documentation build that happens as a cron job on Jenkins.

The error log at
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/690/consoleFull
is a run that happened after this PR and
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/689/consoleFull
is a run from before

It looks like this PR somehow introduces a dependency on netty 3.2.2
through mqtt, curator etc.

Note: Unresolved dependencies path:�
org.jboss.netty:netty:3.2.2.Final�
+-org.apache.zookeeper:zookeeper:3.4.5 ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.curator:curator-client:2.4.0�
+- org.apache.curator:curator-framework:2.4.0�
+- org.apache.curator:curator-recipes:2.4.0 ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.spark:spark-core_2.10:1.5.0-SNAPSHOT�
+- org.apache.spark:spark-streaming_2.10:1.5.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.spark:spark-streaming-mqtt_2.10:1.5.0-SNAPSHOT
+- org.apache.spark:spark-streaming-mqtt-assembly_2.10:1.5.0-SNAPSHOT�

Did some exclude rules change or something like that ?

cc @rxin https://github.com/rxin

—
Reply to this email directly or view it on GitHub
#8069 (comment).

tdas · 2015-08-19T07:51:47Z

Workers that are already fixed are (along with a successfully compiled
build):

worker-02:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3288/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.3,label=spark-test/console
worker-06:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3288/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=spark-test/console
worker-08:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3288/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.0,label=spark-test/console

On Wed, Aug 19, 2015 at 12:43 AM, Tathagata Das [email protected] wrote:

The issues is actually SBT failing to resolve the artifact in the m2 cache
even though the artifact is present. Basically the ivy2 cache of jboss
needs to blown away.

I am slowly fixing the issue on the jenkins workers. Since I am not able
to access the Jenkins workers directly (ssh access not working for me), I
am leveraging the Spark SBT master build to blow away cached jboss. And
whatever worker it is running, that worker is getting fixed. Here is what
fixes it.

rm -rf /home/jenkins/.ivy2/cache/org.jboss*
rm -rf /home/jenkins/.ivy2/cache/junit*

I have it added in the Spark-SBT-master -
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/configure
If you can login into the workers, would you mind running these two lines
on each of them?

On Wed, Aug 19, 2015 at 12:16 AM, Shivaram Venkataraman <
[email protected]> wrote:

@srowen https://github.com/srowen @tdas https://github.com/tdas It
looks like this PR somehow changed the dependency tree and affected the
documentation build that happens as a cron job on Jenkins.

The error log at
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/690/consoleFull
is a run that happened after this PR and
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/job/Spark-Master-Docs/689/consoleFull
is a run from before

It looks like this PR somehow introduces a dependency on netty 3.2.2
through mqtt, curator etc.

Note: Unresolved dependencies path:�
org.jboss.netty:netty:3.2.2.Final�
+-org.apache.zookeeper:zookeeper:3.4.5 ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.curator:curator-client:2.4.0�
+- org.apache.curator:curator-framework:2.4.0�
+- org.apache.curator:curator-recipes:2.4.0 ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.spark:spark-core_2.10:1.5.0-SNAPSHOT�
+- org.apache.spark:spark-streaming_2.10:1.5.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper)
+- org.apache.spark:spark-streaming-mqtt_2.10:1.5.0-SNAPSHOT
+- org.apache.spark:spark-streaming-mqtt-assembly_2.10:1.5.0-SNAPSHOT�

Did some exclude rules change or something like that ?

cc @rxin https://github.com/rxin

—
Reply to this email directly or view it on GitHub
#8069 (comment).

shivaram · 2015-08-19T08:03:51Z

Yeah I have ssh access. Let me give the other ones a go

shivaram · 2015-08-19T08:05:51Z

Well I have ssh access but no permission to delete these files. I think your approach will clear this out or I'll ping Shane tomorrow

zsxwing added 3 commits August 10, 2015 23:42

Demote dependencies already included in the Spark assembly for stream…

5ed1397

…ing-flume-assembly

Demote dependencies already included in the Spark assembly for stream…

50d246a

…ing-kafka-assembly

Demote dependencies already included in the Spark assembly for stream…

9e06559

…ing-kinesis-asl-assembly

zsxwing added 2 commits August 11, 2015 11:32

Merge branch 'master' into SPARK-9574

750218f

Demote dependencies already included in the Spark assembly for stream…

546450a

…ing-mqtt-assembly

tdas reviewed Aug 11, 2015
View reviewed changes

asfgit closed this in bf1d661 Aug 18, 2015

zsxwing deleted the SPARK-9574 branch August 18, 2015 23:44

[SPARK-9574][Streaming]Remove unnecessary contents of spark-streaming-XXX-assembly jars #8069

[SPARK-9574][Streaming]Remove unnecessary contents of spark-streaming-XXX-assembly jars #8069

Uh oh!

Conversation

zsxwing commented Aug 10, 2015

Uh oh!

srowen commented Aug 10, 2015

Uh oh!

SparkQA commented Aug 10, 2015

Uh oh!

tdas commented Aug 10, 2015

Uh oh!

SparkQA commented Aug 10, 2015

Uh oh!

tdas commented Aug 10, 2015

Uh oh!

tdas Aug 11, 2015

Choose a reason for hiding this comment

Uh oh!

tdas Aug 11, 2015

Choose a reason for hiding this comment

Uh oh!

zsxwing Aug 11, 2015

Choose a reason for hiding this comment

Uh oh!

zsxwing Aug 11, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

zsxwing commented Aug 12, 2015

Uh oh!

SparkQA commented Aug 18, 2015

Uh oh!

zsxwing commented Aug 18, 2015

Uh oh!

SparkQA commented Aug 18, 2015

Uh oh!

tdas commented Aug 18, 2015

Uh oh!

shivaram commented Aug 19, 2015

Uh oh!

tdas commented Aug 19, 2015

Uh oh!

tdas commented Aug 19, 2015

Uh oh!

shivaram commented Aug 19, 2015

Uh oh!

shivaram commented Aug 19, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants