Skip to content

[SPARK-33084][CORE][SQL] Add jar support ivy path #29966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 57 commits into from
Closed

[SPARK-33084][CORE][SQL] Add jar support ivy path #29966

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
afaf7bd
[SPARK-33084][CORE][SQL]Add jar support ivy path
AngersZhuuuu Oct 7, 2020
51daf9a
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Oct 7, 2020
3579de0
Update SparkContextSuite.scala
AngersZhuuuu Oct 7, 2020
d6e8caf
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Oct 7, 2020
169e1f8
Update Utils.scala
AngersZhuuuu Oct 8, 2020
0e589ec
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Nov 21, 2020
b3e3211
resolve conflict
AngersZhuuuu Nov 23, 2020
9161340
Update SessionState.scala
AngersZhuuuu Nov 23, 2020
0e3c1ec
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Nov 24, 2020
300ca56
follow comment
AngersZhuuuu Nov 24, 2020
63e877b
https://github.com/apache/spark/pull/29966#discussion_r529242208
AngersZhuuuu Nov 24, 2020
733e62c
follow comment
AngersZhuuuu Nov 24, 2020
b60ba1e
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Nov 24, 2020
883b9d3
fix uri re-use
AngersZhuuuu Nov 28, 2020
208afc2
follow comment
AngersZhuuuu Nov 28, 2020
ba9ea29
add warn message whe multiple trasitive
AngersZhuuuu Nov 28, 2020
10b3737
move DependencyUtils
AngersZhuuuu Nov 28, 2020
7f878c2
add ut
AngersZhuuuu Nov 28, 2020
d2c1950
Update SessionState.scala
AngersZhuuuu Nov 28, 2020
2200076
Update SessionState.scala
AngersZhuuuu Nov 28, 2020
5a9cc30
Update DependencyUtils.scala
AngersZhuuuu Nov 29, 2020
875d8a7
Add end to end test
AngersZhuuuu Nov 29, 2020
e921245
Update SQLQuerySuite.scala
AngersZhuuuu Nov 29, 2020
614a865
follow comment
AngersZhuuuu Nov 30, 2020
8c5cb7c
Update SparkContext.scala
AngersZhuuuu Nov 30, 2020
f460974
fix local path with comma
AngersZhuuuu Nov 30, 2020
1f7dc01
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Dec 1, 2020
050c410
follow comment
AngersZhuuuu Dec 1, 2020
ff611a6
Update SessionState.scala
AngersZhuuuu Dec 1, 2020
03aca3b
split UT and use simply dependency ivy path
AngersZhuuuu Dec 1, 2020
653b919
Update DependencyUtils.scala
AngersZhuuuu Dec 1, 2020
6e48275
Update SparkContext.scala
AngersZhuuuu Dec 1, 2020
bdc5035
follow comment
AngersZhuuuu Dec 2, 2020
9c22882
follow comment
AngersZhuuuu Dec 2, 2020
9c88f8d
follow comment
AngersZhuuuu Dec 2, 2020
8220e5a
Update SparkContextSuite.scala
AngersZhuuuu Dec 2, 2020
49ac62c
follow comment
AngersZhuuuu Dec 2, 2020
b69a62e
Update DependencyUtils.scala
AngersZhuuuu Dec 2, 2020
273a5ac
Follow comment
AngersZhuuuu Dec 3, 2020
ebe1c9c
Update DependencyUtils.scala
AngersZhuuuu Dec 4, 2020
6034fb2
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Dec 5, 2020
e22e398
Update SparkContext.scala
AngersZhuuuu Dec 7, 2020
afea73f
Update SparkContext.scala
AngersZhuuuu Dec 7, 2020
13000f2
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Dec 14, 2020
bce3d40
Update SparkContext.scala
AngersZhuuuu Dec 14, 2020
d53f302
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Dec 14, 2020
57c351d
Update HiveQuerySuite.scala
AngersZhuuuu Dec 15, 2020
8c53b83
follow comment
AngersZhuuuu Dec 22, 2020
4048c5b
https://github.com/apache/spark/pull/29966#discussion_r547040115
AngersZhuuuu Dec 22, 2020
aa53482
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Dec 22, 2020
2ffb431
Update SQLQuerySuite.scala
AngersZhuuuu Dec 22, 2020
8c18cdf
Update SparkContext.scala
AngersZhuuuu Dec 23, 2020
6bd41cd
Update SparkSubmit.scala
AngersZhuuuu Dec 23, 2020
fbc236c
follwo comment
AngersZhuuuu Dec 23, 2020
90491d5
Update DependencyUtils.scala
AngersZhuuuu Dec 23, 2020
75ff3ce
Update SparkContextSuite.scala
AngersZhuuuu Dec 23, 2020
4c44dae
follow comment remove default value
AngersZhuuuu Dec 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
move DependencyUtils
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Nov 28, 2020
commit 10b373783bc5015ff9e5fc01b53ad7d0bde02c01
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1923,7 +1923,7 @@ class SparkContext(config: SparkConf) extends Logging {
case "ivy" =>
Copy link
Member

@HyukjinKwon HyukjinKwon Dec 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AngersZhuuuu, out of curiosity, is the Ivy URI the standard form documented somewhere? or something specific to Spark that you came up with?

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Dec 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AngersZhuuuu, out of curiosity, is the Ivy URI the standard form documented somewhere? or something specific to Spark that you came up with?

From hive https://issues.apache.org/jira/browse/HIVE-9664, since it download jar use ivy then use schema as ivy? I think this useful for a lot of companies that have standard package management, so I implemented it in Spark

// Since `new Path(path).toUri` will lose query information,
// so here we use `URI.create(path)`
Utils.resolveMavenDependencies(URI.create(path))
DependencyUtils.resolveMavenDependencies(URI.create(path))
Copy link
Member

@maropu maropu Dec 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if two added jars have the same dependency with different versions? e.g.,

sc.addJar("ivy://lib1:1.0?transitive=true") // --> it depends on `libX v1.0`
sc.addJar("ivy://lib2:1.0?transitive=true") // --> it depends on `libX v2.0`

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Dec 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

case _ => checkRemoteJarFile(path)
}
}
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.io.File
import org.apache.commons.lang3.StringUtils

import org.apache.spark.{SecurityManager, SparkConf}
import org.apache.spark.deploy.{DependencyUtils, SparkHadoopUtil}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{config, Logging}
import org.apache.spark.rpc.RpcEnv
import org.apache.spark.util._
Expand Down
14 changes: 8 additions & 6 deletions ...apache/spark/deploy/DependencyUtils.scala → ...g/apache/spark/util/DependencyUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* limitations under the License.
*/

package org.apache.spark.deploy
package org.apache.spark.util

import java.io.File
import java.net.URI
Expand All @@ -25,10 +25,10 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}

import org.apache.spark.{SecurityManager, SparkConf, SparkException}
import org.apache.spark.deploy.SparkSubmitUtils
import org.apache.spark.internal.Logging
import org.apache.spark.util.{MutableURLClassLoader, Utils}

private[deploy] object DependencyUtils extends Logging {
private[spark] object DependencyUtils extends Logging {

def getIvyProperties(): Seq[String] = {
Copy link
Contributor

@xkrogen xkrogen Dec 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seq[String] seems like an odd return value for this. Maybe a tuple would be more appropriate? Or even a case class? For either seq or tuple I think we need to add some Scaladoc to tell callers how to unpack it.

Also I wonder if we should be returning Option[String] instead of null for missing values?

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Dec 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add case class is good and then we don't need docs since parameter name is clear enough.
But change to Option[String] will change a lot related function, I think we can refactor this later like #29966 (comment)

Copy link
Contributor

@xkrogen xkrogen Dec 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.

Seq(
Expand All @@ -46,7 +46,8 @@ private[deploy] object DependencyUtils extends Logging {
Array.empty[String]
} else {
val mapTokens = queryString.split("&")
assert(mapTokens.forall(_.split("=").length == 2), "Invalid query string: " + queryString)
assert(mapTokens.forall(_.split("=").length == 2)
, "Invalid URI query string: [ " + queryString + " ]")
mapTokens.map(_.split("=")).map(kv => (kv(0), kv(1))).filter(_._1 == queryTag).map(_._2)
}
}
Expand All @@ -65,7 +66,8 @@ private[deploy] object DependencyUtils extends Logging {
.flatMap { excludeString =>
val excludes: Array[String] = excludeString.split(",")
assert(excludes.forall(_.split(":").length == 2),
"Invalid exclude string: expected 'org:module,org:module,..', found " + excludeString)
"Invalid exclude string: expected 'org:module,org:module,..'," +
" found [ " + excludeString + " ]")
excludes
}.mkString(":")
}
Expand Down Expand Up @@ -106,7 +108,7 @@ private[deploy] object DependencyUtils extends Logging {
uri.getAuthority,
repositories,
ivyRepoPath,
Some(ivySettingsPath)
Option(ivySettingsPath)
)
}

Expand Down
2 changes: 1 addition & 1 deletion core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.UI._
import org.apache.spark.launcher.SparkLauncher
import org.apache.spark.util.{CommandLineUtils, ResetSystemProperties, Utils}
import org.apache.spark.util.{CommandLineUtils, DependencyUtils, ResetSystemProperties, Utils}

trait TestPrematureExit {
suite: SparkFunSuite =>
Expand Down