You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
### What changes were proposed in this pull request?
This PR aims to add a new profile, `huaweicloud-provided`.
### Why are the changes needed?
Since Apache Spark 4.0.0, Apache Spark module is moving from `OkHttp` to `Vert.x` implementation via the following.
- #49159
- [SPARK-37687 Cleanup direct usage of OkHttpClient](https://issues.apache.org/jira/browse/SPARK-37687)
Like Apache Hadoop community, we are moving away further from `okhttp` transitive dependencies from `hadoop-huaweicloud` dependency.
- [HADOOP-18503](https://issues.apache.org/jira/browse/HADOOP-18503) Upgrade Huawei OBS client to 3.22.3.1
- [HADOOP-18890](https://issues.apache.org/jira/browse/HADOOP-18890) Remove okhttp usage
This PR will allow users to exclude and add their `huaweicloud` and its transitive dependencies. Technically, the scope of following dependencies are changed to `provided`. As a result, those are removed from Spark distribution.
```
-esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
-hadoop-huaweicloud/3.4.2//hadoop-huaweicloud-3.4.2.jar
-java-xmlbuilder/1.2//java-xmlbuilder-1.2.jar
-okhttp/3.12.12//okhttp-3.12.12.jar
-okio/1.17.6//okio-1.17.6.jar
```
### Does this PR introduce _any_ user-facing change?
No, this is a new profile which is disabled by default.
### How was this patch tested?
Manually check like the following.
```
$ mvn dependency:tree -Phadoop-cloud | grep okhttp
[INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile
[INFO] | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile
$ mvn dependency:tree -Phadoop-cloud -Phuaweicloud-provided | grep okhttp
[INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:provided
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#52346 from dongjoon-hyun/SPARK-53590.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
0 commit comments