unify SparkApplication defaulting logic

[zhzhuang-zju]

Purpose of this PR

This PR consolidates the SparkApplication defaulting logic that was previously maintained in two separate places (API declaration level and webhook-specific defaults) into a unified approach using Kubernetes SchemeBuilder pattern.

Proposed changes:

• Unified SparkApplication defaulting logic by moving all default value setting to the API layer (api/v1beta2/defaults.go)
• Integrated defaulting functions with SchemeBuilder using runtime.NewSchemeBuilder(addKnownTypes, addDefaultingFuncs)
• Simplified webhook implementation to use operatorscheme.WebhookScheme.Default(app) for consistent default application

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Previously, SparkApplication default values were maintained in two different locations, which created potential inconsistencies and maintenance overhead. By consolidating the defaulting logic into the SchemeBuilder pattern:

1. Single source of truth: All default values are now defined in one place
2. Consistency: Ensures the same defaults are applied regardless of how SparkApplication objects are created or processed
3. Maintainability: Reduces code duplication and makes it easier to maintain and update default values

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

This change maintains backward compatibility while improving the internal architecture. The webhook behavior remains the same from an external perspective.

[zhzhuang-zju]

Verification Report：

1. Set up Spark Operator

$ kubectl get pods --namespace spark-operator 
NAME                                         READY   STATUS    RESTARTS   AGE
spark-operator-controller-74b96777cd-jwcnn   1/1     Running   0          45m
spark-operator-webhook-996c9c77b-9mmpg       1/1     Running   0          45m

2. Deploy a sparksapplication without setting sepc.executor.instance

$ kubectl apply -f - <<EOF
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: docker.io/library/spark:4.0.0
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  arguments:
    - "5000"
  sparkVersion: 4.0.0
  driver:
    labels:
      version: 4.0.0
    memory: 512m
    memoryOverhead: 100m
    serviceAccount: spark-operator-spark
    securityContext:
      capabilities:
        drop:
          - ALL
      runAsGroup: 185
      runAsUser: 185
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
  executor:
    labels:
      version: 4.0.0
    cores: 1
    memory: 512m
    securityContext:
      capabilities:
        drop:
          - ALL
      runAsGroup: 185
      runAsUser: 185
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
EOF

3. spec.executor.instances is set to 1.

$ kubectl get sparkapplications.sparkoperator.k8s.io spark-pi -oyaml |grep instance
    instances: 1

[ChenYi015]

The currecnt CI workflow is broken, we will fix it and then review this PR again.

[ChenYi015]

/ok-to-test

[ChenYi015]

@zhzhuang-zju Thank you for your contribution for improving the consistency of defaulting logic.
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment