[SPARK-40699][DOCS] Supplement undocumented yarn configurations in documentation

[dcoliversun]

What changes were proposed in this pull request?

This PR aims to supplement undocumented yarn configuration in documentation.

Why are the changes needed?

Help users to confirm yarn configurations through documentation instead of code.

Does this PR introduce any user-facing change?

Yes, more configurations in documentation.

How was this patch tested?

Pass the GA.

[dcoliversun]

cc @HyukjinKwon @dongjoon-hyun
It would be good if you have a time to review this PR

[dcoliversun]

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala

Lines 81 to 96 in c4cd193

	private[spark] val AM_TOKEN_CONF_REGEX =
	ConfigBuilder("spark.yarn.am.tokenConfRegex")
	.doc("This config is only supported when Hadoop version is 2.9+ or 3.x (e.g., when using " +
	"the Hadoop 3.x profile). The value of this config is a regex expression used to grep a " +
	"list of config entries from the job's configuration file (e.g., hdfs-site.xml) and send " +
	"to RM, which uses them when renewing delegation tokens. A typical use case of this " +
	"feature is to support delegation tokens in an environment where a YARN cluster needs to " +
	"talk to multiple downstream HDFS clusters, where the YARN RM may not have configs " +
	"(e.g., dfs.nameservices, dfs.ha.namenodes.*, dfs.namenode.rpc-address.*) to connect to " +
	"these clusters. In this scenario, Spark users can specify the config value to be " +
	"'^dfs.nameservices$|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$' to parse " +
	"these HDFS configs from the job's local configuration files. This config is very " +
	"similar to 'mapreduce.job.send-token-conf'. Please check YARN-5910 for more details.")
	.version("3.3.0")
	.stringConf
	.createOptional

[dcoliversun]

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala

Lines 228 to 233 in c4cd193

	private[spark] val CLIENT_LAUNCH_MONITOR_INTERVAL =
	ConfigBuilder("spark.yarn.clientLaunchMonitorInterval")
	.doc("Interval between requests for status the client mode AM when starting the app.")
	.version("2.3.0")
	.timeConf(TimeUnit.MILLISECONDS)
	.createWithDefaultString("1s")

[dcoliversun]

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala

Lines 235 to 242 in c4cd193

	private[spark] val CLIENT_INCLUDE_DRIVER_LOGS_LINK =
	ConfigBuilder("spark.yarn.includeDriverLogsLink")
	.doc("In cluster mode, whether the client application report includes links to the driver "
	+ "container's logs. This requires polling the ResourceManager's REST API, so it "
	+ "places some additional load on the RM.")
	.version("3.1.0")
	.booleanConf
	.createWithDefault(false)

[dcoliversun]

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala

Lines 346 to 351 in c4cd193

	private[spark] val YARN_UNMANAGED_AM = ConfigBuilder("spark.yarn.unmanagedAM.enabled")
	.doc("In client mode, whether to launch the Application Master service as part of the client " +
	"using unmanaged am.")
	.version("3.0.0")
	.booleanConf
	.createWithDefault(false)

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Merged to master