Skip to content

[SPARK-33084][CORE][SQL] Add jar support ivy path #29966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 57 commits into from
Closed

[SPARK-33084][CORE][SQL] Add jar support ivy path #29966

Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
afaf7bd
[SPARK-33084][CORE][SQL]Add jar support ivy path
AngersZhuuuu Oct 7, 2020
51daf9a
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Oct 7, 2020
3579de0
Update SparkContextSuite.scala
AngersZhuuuu Oct 7, 2020
d6e8caf
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Oct 7, 2020
169e1f8
Update Utils.scala
AngersZhuuuu Oct 8, 2020
0e589ec
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Nov 21, 2020
b3e3211
resolve conflict
AngersZhuuuu Nov 23, 2020
9161340
Update SessionState.scala
AngersZhuuuu Nov 23, 2020
0e3c1ec
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Nov 24, 2020
300ca56
follow comment
AngersZhuuuu Nov 24, 2020
63e877b
https://github.com/apache/spark/pull/29966#discussion_r529242208
AngersZhuuuu Nov 24, 2020
733e62c
follow comment
AngersZhuuuu Nov 24, 2020
b60ba1e
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Nov 24, 2020
883b9d3
fix uri re-use
AngersZhuuuu Nov 28, 2020
208afc2
follow comment
AngersZhuuuu Nov 28, 2020
ba9ea29
add warn message whe multiple trasitive
AngersZhuuuu Nov 28, 2020
10b3737
move DependencyUtils
AngersZhuuuu Nov 28, 2020
7f878c2
add ut
AngersZhuuuu Nov 28, 2020
d2c1950
Update SessionState.scala
AngersZhuuuu Nov 28, 2020
2200076
Update SessionState.scala
AngersZhuuuu Nov 28, 2020
5a9cc30
Update DependencyUtils.scala
AngersZhuuuu Nov 29, 2020
875d8a7
Add end to end test
AngersZhuuuu Nov 29, 2020
e921245
Update SQLQuerySuite.scala
AngersZhuuuu Nov 29, 2020
614a865
follow comment
AngersZhuuuu Nov 30, 2020
8c5cb7c
Update SparkContext.scala
AngersZhuuuu Nov 30, 2020
f460974
fix local path with comma
AngersZhuuuu Nov 30, 2020
1f7dc01
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Dec 1, 2020
050c410
follow comment
AngersZhuuuu Dec 1, 2020
ff611a6
Update SessionState.scala
AngersZhuuuu Dec 1, 2020
03aca3b
split UT and use simply dependency ivy path
AngersZhuuuu Dec 1, 2020
653b919
Update DependencyUtils.scala
AngersZhuuuu Dec 1, 2020
6e48275
Update SparkContext.scala
AngersZhuuuu Dec 1, 2020
bdc5035
follow comment
AngersZhuuuu Dec 2, 2020
9c22882
follow comment
AngersZhuuuu Dec 2, 2020
9c88f8d
follow comment
AngersZhuuuu Dec 2, 2020
8220e5a
Update SparkContextSuite.scala
AngersZhuuuu Dec 2, 2020
49ac62c
follow comment
AngersZhuuuu Dec 2, 2020
b69a62e
Update DependencyUtils.scala
AngersZhuuuu Dec 2, 2020
273a5ac
Follow comment
AngersZhuuuu Dec 3, 2020
ebe1c9c
Update DependencyUtils.scala
AngersZhuuuu Dec 4, 2020
6034fb2
Update sql-ref-syntax-aux-resource-mgmt-add-jar.md
AngersZhuuuu Dec 5, 2020
e22e398
Update SparkContext.scala
AngersZhuuuu Dec 7, 2020
afea73f
Update SparkContext.scala
AngersZhuuuu Dec 7, 2020
13000f2
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Dec 14, 2020
bce3d40
Update SparkContext.scala
AngersZhuuuu Dec 14, 2020
d53f302
Merge branch 'support-add-jar-ivy' of https://github.com/AngersZhuuuu…
AngersZhuuuu Dec 14, 2020
57c351d
Update HiveQuerySuite.scala
AngersZhuuuu Dec 15, 2020
8c53b83
follow comment
AngersZhuuuu Dec 22, 2020
4048c5b
https://github.com/apache/spark/pull/29966#discussion_r547040115
AngersZhuuuu Dec 22, 2020
aa53482
Merge branch 'master' into support-add-jar-ivy
AngersZhuuuu Dec 22, 2020
2ffb431
Update SQLQuerySuite.scala
AngersZhuuuu Dec 22, 2020
8c18cdf
Update SparkContext.scala
AngersZhuuuu Dec 23, 2020
6bd41cd
Update SparkSubmit.scala
AngersZhuuuu Dec 23, 2020
fbc236c
follwo comment
AngersZhuuuu Dec 23, 2020
90491d5
Update DependencyUtils.scala
AngersZhuuuu Dec 23, 2020
75ff3ce
Update SparkContextSuite.scala
AngersZhuuuu Dec 23, 2020
4c44dae
follow comment remove default value
AngersZhuuuu Dec 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update HiveQuerySuite.scala
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Dec 15, 2020
commit 57c351d076da51fd8c41323f16ff8d8fbc4b964b
16 changes: 16 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1219,6 +1219,22 @@ class HiveQuerySuite extends HiveComparisonTest with SQLTestUtils with BeforeAnd
}
}
}

test("SPARK-33084: Add jar support ivy url in SQL") {
Copy link
Member

@maropu maropu Dec 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add tests for the same purpose in sql/core, too?

Copy link
Contributor Author

@AngersZhuuuu AngersZhuuuu Dec 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add tests for the same purpose in sql/core, too?

Add UT test("SPARK-33084: Add jar support Ivy URI in SQL -- jar contains udf class") 

class Spark33084 extends UserDefinedAggregateFunction {
  // Data types of input arguments of this aggregate function
  def inputSchema: StructType = StructType(StructField("inputColumn", LongType) :: Nil)
  // Data types of values in the aggregation buffer
  def bufferSchema: StructType = {
    StructType(StructField("sum", LongType) :: StructField("count", LongType) :: Nil)
  }
  // The data type of the returned value
  def dataType: DataType = DoubleType
  // Whether this function always returns the same output on the identical input
  def deterministic: Boolean = true
  // Initializes the given aggregatPARKion buffer. The buffer itself is a `Row` that in addition to
  // standard methods like retrieving a value at an index (e.g., get(), getBoolean()), provides
  // the opportunity to update its values. Note that arrays and maps inside the buffer are still
  // immutable.
  def initialize(buffer: MutableAggregationBuffer): Unit = {
    buffer(0) = 0L
    buffer(1) = 0L
  }
  // Updates the given aggregation buffer `buffer` with new input data from `input`
  def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
    if (!input.isNullAt(0)) {
      buffer(0) = buffer.getLong(0) + input.getLong(0)
      buffer(1) = buffer.getLong(1) + 1
    }
  }
  // Merges two aggregation buffers and stores the updated buffer values back to `buffer1`
  def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
    buffer1(0) = buffer1.getLong(0) + buffer2.getLong(0)
    buffer1(1) = buffer1.getLong(1) + buffer2.getLong(1)
  }
  // Calculates the final result
  def evaluate(buffer: Row): Double = buffer.getLong(0).toDouble / buffer.getLong(1)
}

val testData = TestHive.getHiveFile("data/files/sample.json").toURI
sql("ADD JAR ivy://org.apache.hive.hcatalog:hive-hcatalog-core:2.3.7")
sql(
"""CREATE TABLE t1(a string, b string)
|ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'""".stripMargin)
sql(s"""LOAD DATA LOCAL INPATH "$testData" INTO TABLE t1""")
sql("select * from src join t1 on src.key = t1.a")
sql("DROP TABLE t1")
assert(sql("list jars").filter(_.getString(0).contains(
"org.apache.hive.hcatalog_hive-hcatalog-core-2.3.7.jar")).count() > 0)
assert(sql("list jar").
filter(_.getString(0).contains(
"org.apache.hive.hcatalog_hive-hcatalog-core-2.3.7.jar")).count() > 0)
}
}

// for SPARK-2180 test
Expand Down