Skip to content

Commit e3dce42

Browse files
committed
[SPARK-53590][BUILD] Add huaweicloud-provided profile
### What changes were proposed in this pull request? This PR aims to add a new profile, `huaweicloud-provided`. ### Why are the changes needed? Since Apache Spark 4.0.0, Apache Spark module is moving from `OkHttp` to `Vert.x` implementation via the following. - #49159 - [SPARK-37687 Cleanup direct usage of OkHttpClient](https://issues.apache.org/jira/browse/SPARK-37687) Like Apache Hadoop community, we are moving away further from `okhttp` transitive dependencies from `hadoop-huaweicloud` dependency. - [HADOOP-18503](https://issues.apache.org/jira/browse/HADOOP-18503) Upgrade Huawei OBS client to 3.22.3.1 - [HADOOP-18890](https://issues.apache.org/jira/browse/HADOOP-18890) Remove okhttp usage This PR will allow users to exclude and add their `huaweicloud` and its transitive dependencies. Technically, the scope of following dependencies are changed to `provided`. As a result, those are removed from Spark distribution. ``` -esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar -hadoop-huaweicloud/3.4.2//hadoop-huaweicloud-3.4.2.jar -java-xmlbuilder/1.2//java-xmlbuilder-1.2.jar -okhttp/3.12.12//okhttp-3.12.12.jar -okio/1.17.6//okio-1.17.6.jar ``` ### Does this PR introduce _any_ user-facing change? No, this is a new profile which is disabled by default. ### How was this patch tested? Manually check like the following. ``` $ mvn dependency:tree -Phadoop-cloud | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile $ mvn dependency:tree -Phadoop-cloud -Phuaweicloud-provided | grep okhttp [INFO] +- com.squareup.okhttp3:okhttp:jar:3.12.12:provided ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52346 from dongjoon-hyun/SPARK-53590. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 6c9e750 commit e3dce42

File tree

2 files changed

+20
-0
lines changed

2 files changed

+20
-0
lines changed

hadoop-cloud/pom.xml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,12 @@
166166
<version>${hadoop.version}</version>
167167
<scope>${hadoop.deps.scope}</scope>
168168
</dependency>
169+
<dependency>
170+
<groupId>org.apache.hadoop</groupId>
171+
<artifactId>hadoop-huaweicloud</artifactId>
172+
<version>${hadoop.version}</version>
173+
<scope>${huaweicloud.deps.scope}</scope>
174+
</dependency>
169175
<!--
170176
There's now a hadoop-cloud-storage which transitively pulls in the store JARs,
171177
but it still needs some selective exclusion across versions, especially 3.0.x.
@@ -191,6 +197,10 @@
191197
<groupId>org.apache.hadoop</groupId>
192198
<artifactId>hadoop-cos</artifactId>
193199
</exclusion>
200+
<exclusion>
201+
<groupId>org.apache.hadoop</groupId>
202+
<artifactId>hadoop-huaweicloud</artifactId>
203+
</exclusion>
194204
</exclusions>
195205
</dependency>
196206
<!--
@@ -215,11 +225,13 @@
215225
<groupId>com.squareup.okhttp3</groupId>
216226
<artifactId>okhttp</artifactId>
217227
<version>${okhttp.version}</version>
228+
<scope>${huaweicloud.deps.scope}</scope>
218229
</dependency>
219230
<dependency>
220231
<groupId>com.squareup.okio</groupId>
221232
<artifactId>okio</artifactId>
222233
<version>${okio.version}</version>
234+
<scope>${huaweicloud.deps.scope}</scope>
223235
</dependency>
224236
</dependencies>
225237

@@ -248,6 +260,12 @@
248260
</build>
249261

250262
<profiles>
263+
<profile>
264+
<id>huaweicloud-provided</id>
265+
<properties>
266+
<huaweicloud.deps.scope>provided</huaweicloud.deps.scope>
267+
</properties>
268+
</profile>
251269
<profile>
252270
<id>integration-test</id>
253271
<build>

pom.xml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,7 @@
279279
-->
280280
<derby.deps.scope>compile</derby.deps.scope>
281281
<hadoop.deps.scope>compile</hadoop.deps.scope>
282+
<huaweicloud.deps.scope>compile</huaweicloud.deps.scope>
282283
<hive.deps.scope>compile</hive.deps.scope>
283284
<hive.storage.version>2.8.1</hive.storage.version>
284285
<hive.storage.scope>compile</hive.storage.scope>
@@ -3526,6 +3527,7 @@
35263527
<id>hadoop-provided</id>
35273528
<properties>
35283529
<spark.yarn.isHadoopProvided>true</spark.yarn.isHadoopProvided>
3530+
<huaweicloud.deps.scope>provided</huaweicloud.deps.scope>
35293531
</properties>
35303532
</profile>
35313533
<profile>

0 commit comments

Comments
 (0)