[SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader #32515

yaooqinn · 2021-05-12T04:09:52Z

What changes were proposed in this pull request?

In yaooqinn/itachi#8, we had a discussion about the current extension injection for the spark session. We've agreed that the current way is not that convenient for both third-party developers and end-users.

It's much simple if third-party developers can provide a resource file that contains default extensions for Spark to load ahead

Why are the changes needed?

better use experience

Does this PR introduce any user-facing change?

no, dev change

How was this patch tested?

new tests

…ader

SparkQA · 2021-05-12T04:30:07Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42942/

SparkQA · 2021-05-12T04:30:09Z

Test build #138420 has finished for PR 32515 at commit baa61a3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

…ader

SparkQA · 2021-05-12T06:23:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42949/

SparkQA · 2021-05-12T06:24:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42949/

yaooqinn · 2021-05-12T07:23:36Z

cc @cloud-fan @MrPowers @HyukjinKwon @maropu thanks

SparkQA · 2021-05-12T07:31:56Z

Test build #138434 has finished for PR 32515 at commit 4e84a6f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 4e84a6f.

SparkQA · 2021-05-12T07:55:01Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42955/

sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala

sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala

SparkQA · 2021-05-12T08:40:59Z

Test build #138443 has finished for PR 32515 at commit 453ad87.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-12T09:19:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42964/

SparkQA · 2021-05-12T09:26:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42964/

SparkQA · 2021-05-12T10:20:35Z

Test build #138428 has finished for PR 32515 at commit ec731fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-12T12:13:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42971/

SparkQA · 2021-05-12T12:20:33Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42971/

SparkQA · 2021-05-12T12:29:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42973/

SparkQA · 2021-05-12T12:29:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42973/

SparkQA · 2021-05-12T17:10:58Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42982/

SparkQA · 2021-05-12T19:36:01Z

Test build #138457 has finished for PR 32515 at commit e9b05cd.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-12T20:36:28Z

Test build #138461 has finished for PR 32515 at commit 9b5d7c2.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-05-12T20:40:44Z

Ur, @yaooqinn . The R failure looks a little suspicious because it's consistent and relevant. It might be a side-effect in terms of test classes. Could you double-check it?

── 1. FAILURE (test_sparkSQL.R:4000:3): catalog APIs, listTables, listColumns, l
take(orderBy(f, "className"), 1)$className not equal to "org.apache.spark.sql.catalyst.expressions.Abs".
1/1 mismatches
x[1]: "noClass"
y[1]: "org.apache.spark.sql.catalyst.expressions.Abs"

yaooqinn · 2021-05-13T02:30:44Z

Ur, @yaooqinn . The R failure looks a little suspicious because it's consistent and relevant. It might be a side-effect in terms of test classes. Could you double-check it?
── 1. FAILURE (test_sparkSQL.R:4000:3): catalog APIs, listTables, listColumns, l
take(orderBy(f, "className"), 1)$className not equal to "org.apache.spark.sql.catalyst.expressions.Abs".
1/1 mismatches
x[1]: "noClass"
y[1]: "org.apache.spark.sql.catalyst.expressions.Abs"

The failure proves this works well at the sparkR side, LOL. I changed the mocked function classname to fix this

sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala

SparkQA · 2021-05-13T03:16:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43002/

SparkQA · 2021-05-13T03:16:47Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43002/

…sionsProvider.scala Co-authored-by: Hyukjin Kwon <[email protected]>

SparkQA · 2021-05-13T03:33:33Z

Test build #138484 has finished for PR 32515 at commit 8f1be0e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-13T03:36:01Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43004/

HyukjinKwon · 2021-05-13T05:11:38Z

Looks okay to me too

SparkQA · 2021-05-13T05:14:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43009/

SparkQA · 2021-05-13T05:20:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43009/

SparkQA · 2021-05-13T06:15:12Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43013/

SparkQA · 2021-05-13T06:51:58Z

Test build #138482 has finished for PR 32515 at commit a79d76e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2021-05-13T08:35:17Z

Thanks for the help ~ @dongjoon-hyun @HyukjinKwon @cloud-fan

+1 for myself. merged to master

SparkQA · 2021-05-13T08:44:04Z

Test build #138489 has finished for PR 32515 at commit ad18acc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-13T09:57:59Z

Test build #138493 has finished for PR 32515 at commit b8b54ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ider` trait ### What changes were proposed in this pull request? This PR aims to remove `Unstable` from `SparkSessionExtensionsProvider` trait from Apache Spark 4.1.0. https://github.com/apache/spark/blob/400e2b29caf43b443f2e0c42aee92c4d0e3e0eb7/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala#L78-L81 ### Why are the changes needed? Since Apache Spark 3.2.0, we provide `SparkSessionExtensionsProvider` trait without any change to allow third-party developers to provide a resource file that contains default extensions. - #32515 Apache Iceberg has been using this since 0.11.0. - https://iceberg.apache.org/docs/1.7.1/spark-configuration/#sql-extensions > Iceberg 0.11.0 and later add an extension module to Spark to add new SQL commands, like CALL for stored procedures or ALTER TABLE ... WRITE ORDERED BY. Although `SparkSessionExtensions` is an unstable developer API, `SparkSessionExtensionsProvider` trait itself and ServiceLoader framework is stable and it's unlikely for Apache Spark to change this `trait`. https://github.com/apache/spark/blob/400e2b29caf43b443f2e0c42aee92c4d0e3e0eb7/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala#L106-L109 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49780 from dongjoon-hyun/SPARK-51073. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLo…

baa61a3

…ader

github-actions bot added the SQL label May 12, 2021

[WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLo…

ec731fc

…ader

yaooqinn added 2 commits May 12, 2021 15:12

add tests

8ab0b79

avoid API change

4e84a6f

yaooqinn changed the title ~~[WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader~~ [SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader May 12, 2021

Revert "avoid API change"

453ad87

This reverts commit 4e84a6f.

HyukjinKwon reviewed May 12, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed May 12, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala Show resolved Hide resolved

yaooqinn added 2 commits May 12, 2021 19:16

add doc and example

1aece6a

indent

459eab0

github-actions bot added the EXAMPLES label May 12, 2021

cloud-fan approved these changes May 12, 2021

View reviewed changes

yaooqinn added 2 commits May 12, 2021 22:52

doc

d5a4037

doc

e9b05cd

typo

a79d76e

HyukjinKwon reviewed May 13, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed May 13, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensionsProvider.scala Outdated Show resolved Hide resolved

yaooqinn and others added 2 commits May 13, 2021 11:18

Update sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExten…

e795f45

…sionsProvider.scala Co-authored-by: Hyukjin Kwon <[email protected]>

Update sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExten…

8f1be0e

…sionsProvider.scala Co-authored-by: Hyukjin Kwon <[email protected]>

yaooqinn added 2 commits May 13, 2021 11:45

fix build

ad18acc

fix doc build

b8b54ea

yaooqinn self-assigned this May 13, 2021

yaooqinn closed this in 5181543 May 13, 2021

yaooqinn deleted the SPARK-35380 branch March 10, 2022 08:16

dongjoon-hyun mentioned this pull request Feb 3, 2025

[SPARK-51073][SQL] Remove Unstable from SparkSessionExtensionsProvider trait #49780

Closed

[SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader #32515

[SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader #32515

Uh oh!

Conversation

yaooqinn commented May 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

yaooqinn commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

Uh oh!

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

SparkQA commented May 12, 2021

Uh oh!

dongjoon-hyun commented May 12, 2021

Uh oh!

yaooqinn commented May 13, 2021

Uh oh!

Uh oh!

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

HyukjinKwon commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

yaooqinn commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

Reviewers

yaooqinn commented May 12, 2021 •

edited

Loading