[SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile #28640

gaborgsomogyi · 2020-05-26T09:27:52Z

What changes were proposed in this pull request?

There is an unnecessary dependency for mssql-jdbc. In this PR I've removed it.

Why are the changes needed?

Unnecessary dependency.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins with the following configuration.

Pass the dependency test.
SBT with Hadoop-3.2 ([SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile #28640 (comment))
Maven with Hadoop-3.2

gaborgsomogyi · 2020-05-26T09:30:53Z

cc @dongjoon-hyun

dongjoon-hyun · 2020-05-26T09:35:22Z

Thank you for making a PR, but it's weird. This is an auto-generated file. Are you sure this is enough?

SparkQA · 2020-05-26T09:36:16Z

Test build #123117 has finished for PR 28640 at commit 4b333b2.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-26T09:36:39Z

You shall not pass. :)

dev/deps/spark-deps-hadoop-3.2-hive-2.3

HeartSaVioR · 2020-05-26T09:40:46Z

Isn't it weird it's only pulled from Hive 2.3? Does Hive 2.3 refer any of mssql?

dongjoon-hyun · 2020-05-26T09:42:44Z

Ya. Right, @HeartSaVioR . This dependency also surprised me at @gaborgsomogyi 's PR. We need to remove this definitely.

gaborgsomogyi · 2020-05-26T09:47:34Z

There is another dep file with hive 2.3 where it doesn't appear. What I can think of is hadoop 3.2...

gaborgsomogyi · 2020-05-26T10:00:12Z

It's weird, coming from spark-yarn.

gaborgsomogyi · 2020-05-26T10:07:51Z

...
[INFO] +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.2.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:3.2.0:compile
[INFO] |     +- org.apache.hadoop:hadoop-yarn-registry:jar:3.2.0:compile
[INFO] |     |  \- commons-daemon:commons-daemon:jar:1.0.13:compile
[INFO] |     +- org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1:compile
[INFO] |     +- org.ehcache:ehcache:jar:3.3.1:compile
[INFO] |     \- com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7:runtime
...

gaborgsomogyi · 2020-05-26T11:11:53Z

This has been added in YARN-3663 but not yet clear whether it's required on the client side.

gaborgsomogyi · 2020-05-26T12:34:06Z

Based on the commit found here the dependency infiltrated on the client side is bug and not feature. We've excluded HikariCP-java7 as well added in the mentioned PR.

dongjoon-hyun · 2020-05-26T15:11:43Z

pom.xml

+          <exclusion>
+            <groupId>com.microsoft.sqlserver</groupId>
+            <artifactId>mssql-jdbc</artifactId>
+          </exclusion>


Thank you so much!

dongjoon-hyun · 2020-05-26T15:13:24Z

Retest this please.

dongjoon-hyun · 2020-05-26T15:15:19Z

Retest this please.

SparkQA · 2020-05-26T16:00:39Z

Test build #123123 has finished for PR 28640 at commit b6d2094.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-26T18:16:20Z

Test build #123128 has finished for PR 28640 at commit b6d2094.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-26T22:02:42Z

Test build #123129 has finished for PR 28640 at commit b6d2094.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Merged to master/3.0.

…rofile ### What changes were proposed in this pull request? There is an unnecessary dependency for `mssql-jdbc`. In this PR I've removed it. ### Why are the changes needed? Unnecessary dependency. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins with the following configuration. - [x] Pass the dependency test. - [x] SBT with Hadoop-3.2 (#28640 (comment)) - [ ] Maven with Hadoop-3.2 Closes #28640 from gaborgsomogyi/SPARK-31821. Authored-by: Gabor Somogyi <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 8f1d774) Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2020-05-27T01:22:20Z

Thank you so much, @gaborgsomogyi .

gaborgsomogyi · 2020-05-27T07:46:16Z

Just wanted to continue but already finished, thanks for the help @dongjoon-hyun !

[SPARK-31821][BUILD]Remove mssql-jdbc dependencies

4b333b2

probot-autolabeler bot added the BUILD label May 26, 2020

dongjoon-hyun reviewed May 26, 2020

View reviewed changes

dev/deps/spark-deps-hadoop-3.2-hive-2.3 Show resolved Hide resolved

gaborgsomogyi changed the title ~~[SPARK-31821][BUILD]Remove mssql-jdbc dependencies~~ [WIP][SPARK-31821][BUILD]Remove mssql-jdbc dependencies May 26, 2020

dongjoon-hyun mentioned this pull request May 26, 2020

[SPARK-31337][SQL]Support MS SQL Kerberos login in JDBC connector #28635

Closed

Add exclude

b6d2094

gaborgsomogyi changed the title ~~[WIP][SPARK-31821][BUILD]Remove mssql-jdbc dependencies~~ [SPARK-31821][BUILD]Remove mssql-jdbc dependencies May 26, 2020

dongjoon-hyun reviewed May 26, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-31821][BUILD]Remove mssql-jdbc dependencies~~ [SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile May 26, 2020

dongjoon-hyun changed the title ~~[SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile~~ [SPARK-31821][BUILD][test-hadoop3.2] Remove mssql-jdbc dependencies from Hadoop 3.2 profile May 26, 2020

dongjoon-hyun changed the title ~~[SPARK-31821][BUILD][test-hadoop3.2] Remove mssql-jdbc dependencies from Hadoop 3.2 profile~~ [SPARK-31821][BUILD][test-hadoop3.2][test-maven] Remove mssql-jdbc dependencies from Hadoop 3.2 profile May 26, 2020

sarutak mentioned this pull request May 26, 2020

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28627

Closed

dongjoon-hyun changed the title ~~[SPARK-31821][BUILD][test-hadoop3.2][test-maven] Remove mssql-jdbc dependencies from Hadoop 3.2 profile~~ [SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile May 27, 2020

dongjoon-hyun approved these changes May 27, 2020

View reviewed changes

dongjoon-hyun closed this in 8f1d774 May 27, 2020

[SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile #28640

[SPARK-31821][BUILD] Remove mssql-jdbc dependencies from Hadoop 3.2 profile #28640

Uh oh!

Conversation

gaborgsomogyi commented May 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gaborgsomogyi commented May 26, 2020

Uh oh!

dongjoon-hyun commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

dongjoon-hyun commented May 26, 2020

Uh oh!

Uh oh!

HeartSaVioR commented May 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 26, 2020

Uh oh!

gaborgsomogyi commented May 26, 2020

Uh oh!

gaborgsomogyi commented May 26, 2020

Uh oh!

gaborgsomogyi commented May 26, 2020

Uh oh!

gaborgsomogyi commented May 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaborgsomogyi commented May 26, 2020

Uh oh!

dongjoon-hyun May 26, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 26, 2020

Uh oh!

dongjoon-hyun commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

SparkQA commented May 26, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 27, 2020

Uh oh!

gaborgsomogyi commented May 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gaborgsomogyi commented May 26, 2020 •

edited

Loading

HeartSaVioR commented May 26, 2020 •

edited

Loading

gaborgsomogyi commented May 26, 2020 •

edited

Loading