[SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include scala-library

[pan3793]

What changes were proposed in this pull request?

Since SPARK-41400, the common/network-yarn module has started to hard depend on Scala, now it causes YARN Resource Manager to fail to start due to missing scala-library.

2025-07-24 09:55:38,369 INFO util.ApplicationClassLoader: classpath: [file:/opt/spark/yarn/spark-4.1.0-SNAPSHOT-yarn-shuffle.jar]
2025-07-24 09:55:38,369 INFO util.ApplicationClassLoader: system classes: [java., javax.accessibility., -javax.activation., javax.activity., javax.annotation., javax.annotation.processing., javax.crypto., javax.ima
geio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax.naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.secur
ity.cert., javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apache.commons.logging.,
 org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml]
2025-07-24 09:55:38,538 INFO yarn.YarnShuffleService: Initializing YARN shuffle service for Spark
2025-07-24 09:55:38,539 WARN containermanager.AuxServices: The Auxiliary Service named 'spark_shuffle' in the configuration is for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWi
thCustomClassLoader which has a name of 'org.apache.spark.network.yarn.YarnShuffleService with custom class loader'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may ha
ve issues unless the refer to the name in the config.
2025-07-24 09:55:38,808 ERROR nodemanager.NodeManager: Error starting NodeManager
java.lang.NoClassDefFoundError: scala/Product
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:176)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	at org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:330)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceInit(AuxiliaryServiceWithCustomClassLoader.java:64)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:475)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:758)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:336)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:501)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:969)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1058)
Caused by: java.lang.ClassNotFoundException: scala.Product
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
	at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
	... 25 more
2025-07-24 09:55:38,815 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at hadoop-worker1.orb.local/192.168.97.6
************************************************************/

Note: now spark-<version>-yarn-shuffle.jar is ~100m, while previously it is ~10m. in common/utils, the Java and Scala code cross references each other, so we can not simply split it into one Java utils and one Scala utils modules, thus it's not easy to make spark-<version>-yarn-shuffle.jar to be scala-free as before.

Why are the changes needed?

Bug fix, recover a broken feature.

Does this PR introduce any user-facing change?

Yes, recover a broken feature.

How was this patch tested?

Tested on a YARN cluster, RM starts successfully after patching.

Note: Spark 4 requires JDK 17 or later, but JDK 17 is not officially supported as of Hadoop 3.4.1. The Hadoop community has been actively working on supporting JDK 17 in recent months, and it almost works fine in 3.4.2.

For reviewers who expect to verify this locally, consider using 3.4.2 RC1 [1]

[1] https://lists.apache.org/thread/f66vj3rj6cpk37gb1jfl2ombq3hltsml

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

cc @hvanhovell @LuciferYang

[LuciferYang]

Note: now spark--yarn-shuffle.jar is ~100m, while previously it is ~10m. in common/utils, the Java and Scala code cross references each other, so we can not simply split it into one Java utils and one Scala utils modules, thus it's not easy to make spark--yarn-shuffle.jar to be scala-free as before.

Do you know why it has become so big?

[pan3793]

Do you know why it has become so big?

@LuciferYang the jar can be produced by

build/mvn -pl common/network-yarn -am clean package -DskipTests -Pyarn

spark-3.3.4-yarn-shuffle.jar - 11M
spark-3.4.4-yarn-shuffle.jar - 68M -- rocksdbjni contributes the most part
spark-3.5.6-yarn-shuffle.jar - 74M -- start consuming `spark-common-utils_2.12:jar` and it's transitive deps
spark-4.0.0-yarn-shuffle.jar - 96M -- start consuming `netty-tcnative-boringssl-static:jar` -- broken
spark-4.1.0-SNAPSHOT-yarn-shuffle.jar - 103M -- this PR

[mridulm]

This is unfortunate, thanks for working on fixing it.
I am also concerned about the size escalation, and wondering if we can break the dependency on scala for ess.

[pan3793]

... if we can break the dependency on scala for ess.

@mridulm This is what I initially expected, but it seems we can't, because the structured logging framework API for Java depends on Scala classes.

spark/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java

Lines 159 to 165 in 5a9929c

	public void info(String msg, Throwable throwable, MDC... mdcs) {
	if (mdcs == null || mdcs.length == 0) {
	slf4jLogger.info(msg, throwable);
	} else if (slf4jLogger.isInfoEnabled()) {
	withLogContext(msg, mdcs, throwable, mt -> slf4jLogger.info(mt.message, mt.throwable));
	}
	}

[LuciferYang]

... if we can break the dependency on scala for ess.

@mridulm This is what I initially expected, but it seems we can't, because the structured logging framework API for Java depends on Scala classes.

spark/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java

Lines 159 to 165 in 5a9929c

	public void info(String msg, Throwable throwable, MDC... mdcs) {
	if (mdcs == null || mdcs.length == 0) {
	slf4jLogger.info(msg, throwable);
	} else if (slf4jLogger.isInfoEnabled()) {
	withLogContext(msg, mdcs, throwable, mt -> slf4jLogger.info(mt.message, mt.throwable));
	}
	}

also cc @gengliangwang and @panbingkun

[dongjoon-hyun]

Thank you so much for reporting and working on this, @pan3793 .

If then, we may want to remove Structured Logging Framework usage in YARN External Shuffle Service module? Technically, it's useless in YARN, isn't it?

it seems we can't, because the structured logging framework API for Java depends on Scala classes.

cc @peter-toth

[pan3793]

@dongjoon-hyun common/network-yarn module depends on common/utils, common/network-common, common/network-shuffle. If we want to make common/network-yarn scala-free, we should purge Structured Logging Framework usage in all of those modules.

[dongjoon-hyun]

Got it.

[dongjoon-hyun]

Wait, Structured Logging Framework is controlled by configuration. Do you mean Structured Logging Framework-disabled code path also requires Scala at runtime, @pan3793 ? In case of spark.log.structuredLogging.enabled=false, we might be free scala-library like before.

[pan3793]

@dongjoon-hyun spark.log.structuredLogging.enabled=false does not work either. The stacktrace pasted in the PR description is produced under disabling structured logging.

In the Structured Logging Framework API, MDC is a Scala case class, which inherits from scala.Product, thus always requires scala-library when this API is called.

spark/common/utils/src/main/java/org/apache/spark/internal/SparkLogger.java

Lines 159 to 165 in 5a9929c

	public void info(String msg, Throwable throwable, MDC... mdcs) {
	if (mdcs == null || mdcs.length == 0) {
	slf4jLogger.info(msg, throwable);
	} else if (slf4jLogger.isInfoEnabled()) {
	withLogContext(msg, mdcs, throwable, mt -> slf4jLogger.info(mt.message, mt.throwable));
	}
	}

spark/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala

Line 78 in 5a9929c

case class MDC(key: LogKey, value: Any) {

spark/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java

Lines 330 to 331 in 5a9929c

	logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'",
	MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace));

[dongjoon-hyun]

Thank you. Let me rephrase in my words. So, the problem is not 'Structured Logging'. Instead, the root cause is the MDC usage which is newly added based on Scala-dependency. Did I understand correctly?

[pan3793]

I think they are the same thing because MDC is part of the structured logging API.

[dongjoon-hyun]

I think they are the same thing because MDC is part of the structured logging API.

😄 What I was thinking is something like following by removing MDC feature in those key modules (YARN external shuffle service depends), @pan3793 . Although it means we are going to lose MDC feature for those modules, I thought it's a few places compared to the whole Spark modules. However, log is still there, isn't it?

 logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'", 
-   MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace)); 
+   metricsNamespace); 

Structured Logging can be considered two parts. The main part is definitely writing into JSON files. MDC is a secondary part only to enrich the log information.

[dongjoon-hyun]

Anyway, it was just my idea. For this YARN External Shuffle service, I'll leave it to you guys because I'm not using this technically. So, feel free to proceed the AS-IS direction, @pan3793 and all.

[pan3793]

@dongjoon-hyun, thanks for your advice, but I'm afraid this violates the design principle of the Structured Logging Framework. I remember during the development of this feature, @gengliangwang strongly recommended replacing all variables with MDC for INFO and higher severity levels. And in the SparkLogger, there is no such API to allow us to do:

 logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'", 
-   MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace)); 
+   metricsNamespace); 

[dongjoon-hyun]

To @pan3793 , it seems that I wan't clear enough in the previous comment. I already walked away from this PR . That was my my idea but I'm also not insisting anything here since the last comment. Literally, you can ignore me (and previous my all comments) on this PR because I don't have any better suggestion as of now. 😄

@dongjoon-hyun, thanks for your advice, but I'm afraid this violates the design principle of the Structured Logging Framework. I remember during the development of this feature, @gengliangwang strongly recommended replacing all variables with MDC for INFO and higher severity levels. And in the SparkLogger, there is no such API to allow us to do:

 logger.info("Registered metrics with Hadoop's DefaultMetricsSystem using namespace '{}'", 
-   MDC.of(LogKeys.SHUFFLE_SERVICE_METRICS_NAMESPACE$.MODULE$, metricsNamespace)); 
+   metricsNamespace); 

[LuciferYang]

Actually, I'm no longer using the External Shuffle service either. But I'm wondering if it's feasible to rewrite MDC and LogKeys in Java. That should enable decoupling, right? They are both in the org.apache.spark.internal package, so they probably don't belong to the public API either.

[pan3793]

@LuciferYang, thanks for your advice. I have made some progress in rewriting MDC and LogKeys from Scala to Java locally, although it's not done yet, I'm optimistic about its feasibility. But the change is non-trivial, at least not fit for branch-4.0. I suggest adopting this PR as a simple fix first, and I will open a PR follows your idea and discuss later. WDYT?

[LuciferYang]

@LuciferYang, thanks for your advice. I have made some progress in rewriting MDC and LogKeys from Scala to Java locally, although it's not done yet, I'm optimistic about its feasibility. But the change is untrivial, at least not fit for branch-4.0. I suggest adopting this PR as a simple fix first, and I will open a PR follows your idea and discuss later. WDYT?

fine to me

[LuciferYang]

After the merge of the pull request at #51775, master should no longer require this patch, but the branch-4.0 should still need this fix, right?

[pan3793]

After SPARK-53064, I tested that YARN ESS can successfully bootstrap and serve simple Spark queries w/o the scala-library, but I can NOT say that YARN ESS is scala-free, because I cannot guarantee that all the code in YARN ESS does not call Scala utility classes.

To achieve the goal, we might need to split the common/utils into 2 modules(one Java, one Scala), and let other common modules only depend on the pure Java utils.

For safety, we can merge this PR to both master and branch-4.0.

[LuciferYang]

Merged into master/branch-4.0. Thanks @pan3793 @dongjoon-hyun and @mridulm

[mridulm]

Thanks for working on this @pan3793 !
I missed reviewing the rewrite for MDC/LogKeys - but it is great to see this dependency getting removed.