[Bug] RayCluster leaks in RayJob when the cluster terminated exceptionally

[anxietymonger]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

Using RayJob with kuberay v1.3.2, when the pod of head node of the RayCluster terminated exceptionally (e.g. OOMKilled, evicted), the cluster got restarted during reconciliation (but losing the submitted ray jobs I presume). During the process, the submitter pod somehow goes into Complete state, leading to a resource leak where the RayCluster and RayJob will continue to be "Running" indefinitely even though there is no ray job at all in the cluster.

RayJob parameters:
backoffLimit = 0, submitterConfig.backoffLimit = 0, restartPolicy = Never

# kubectl get rayjob
NAME                                   JOB STATUS   DEPLOYMENT STATUS   RAY CLUSTER NAME                                        START TIME             END TIME   AGE
2ff7dc07-d00c-490e-a64e-a463f442fe23   RUNNING      Running             2ff7dc07-d00c-490e-a64e-a463f442fe23-raycluster-8htlh   2025-07-24T08:23:57Z              19h
78ff5104-413f-4208-b4d6-7d894a6f17f8   RUNNING      Running             78ff5104-413f-4208-b4d6-7d894a6f17f8-raycluster-tg8r4   2025-07-24T08:24:15Z              19h
a2e91677-229a-49f2-8e2b-0b59d513d694   RUNNING      Running             a2e91677-229a-49f2-8e2b-0b59d513d694-raycluster-6q7c2   2025-07-24T08:24:20Z              19h
a9d6f1cf-afe6-469c-942c-a4fdf94f5271   RUNNING      Running             a9d6f1cf-afe6-469c-942c-a4fdf94f5271-raycluster-vfmr4   2025-07-24T08:24:10Z              19h
c3156a39-625a-4201-9b04-5030b84c6d8d   RUNNING      Running             c3156a39-625a-4201-9b04-5030b84c6d8d-raycluster-f695z   2025-07-24T08:24:17Z              19h
d0571c80-7162-40e4-a0c5-617cd2d11b54   RUNNING      Running             d0571c80-7162-40e4-a0c5-617cd2d11b54-raycluster-tdcq9   2025-07-24T08:24:12Z              19h
# kubectl get raycluster
NAME                                                    DESIRED WORKERS   AVAILABLE WORKERS   CPUS   MEMORY   GPUS   STATUS   AGE
2ff7dc07-d00c-490e-a64e-a463f442fe23-raycluster-8htlh                                         55     500Gi    0      ready    19h
78ff5104-413f-4208-b4d6-7d894a6f17f8-raycluster-tg8r4                                         55     500Gi    0      ready    19h
a2e91677-229a-49f2-8e2b-0b59d513d694-raycluster-6q7c2                                         55     500Gi    0      ready    19h
a9d6f1cf-afe6-469c-942c-a4fdf94f5271-raycluster-vfmr4                                         55     500Gi    0      ready    19h
c3156a39-625a-4201-9b04-5030b84c6d8d-raycluster-f695z                                         55     500Gi    0      ready    19h
d0571c80-7162-40e4-a0c5-617cd2d11b54-raycluster-tdcq9                                         55     500Gi    0      ready    19h
# kubectl get pod
NAME                                                            READY   STATUS      RESTARTS   AGE
2ff7dc07-d00c-490e-a64e-a463f442fe23-gszzs                      0/1     Completed   0          19h
2ff7dc07-d00c-490e-a64e-a463f442fe23-raycluster-8h-head-44hhb   1/1     Running     0          19h
78ff5104-413f-4208-b4d6-7d894a6f17f8-raycluster-tg-head-6kf6z   1/1     Running     0          19h
78ff5104-413f-4208-b4d6-7d894a6f17f8-xb9k9                      0/1     Completed   0          19h
a2e91677-229a-49f2-8e2b-0b59d513d694-k49pd                      0/1     Completed   0          19h
a2e91677-229a-49f2-8e2b-0b59d513d694-raycluster-6q-head-llhvf   1/1     Running     0          19h
a9d6f1cf-afe6-469c-942c-a4fdf94f5271-27n5b                      0/1     Completed   0          19h
a9d6f1cf-afe6-469c-942c-a4fdf94f5271-raycluster-vf-head-7rltl   1/1     Running     0          19h
c3156a39-625a-4201-9b04-5030b84c6d8d-2l5mb                      0/1     Completed   0          19h
c3156a39-625a-4201-9b04-5030b84c6d8d-raycluster-f6-head-dd6lv   1/1     Running     0          19h
d0571c80-7162-40e4-a0c5-617cd2d11b54-4vsqv                      0/1     Completed   0          19h
d0571c80-7162-40e4-a0c5-617cd2d11b54-raycluster-td-head-54925   1/1     Running     0          19h

Reproduction script

NA

Anything else

Is this because the reconciler only checks the failed condition? Shouldn't it also check other conditions as well to prevent the leak?

kuberay/ray-operator/controllers/ray/rayjob_controller.go

Lines 916 to 935 in ea0b9c5

	func checkK8sJobAndUpdateStatusIfNeeded(ctx context.Context, rayJob *rayv1.RayJob, job *batchv1.Job) bool {
	logger := ctrl.LoggerFrom(ctx)
	for _, cond := range job.Status.Conditions {
	if cond.Type == batchv1.JobFailed && cond.Status == corev1.ConditionTrue {
	logger.Info("The submitter Kubernetes Job has failed. Attempting to transition the status to `Failed`.", "Submitter K8s Job", job.Name, "Reason", cond.Reason, "Message", cond.Message)
	rayJob.Status.JobDeploymentStatus = rayv1.JobDeploymentStatusFailed
	// The submitter Job needs to wait for the user code to finish and retrieve its logs.
	// Therefore, a failed Submitter Job indicates that the submission itself has failed or the user code has thrown an error.
	// If the failure is due to user code, the JobStatus and Job message will be updated accordingly from the previous reconciliation.
	if rayJob.Status.JobStatus == rayv1.JobStatusFailed {
	rayJob.Status.Reason = rayv1.AppFailed
	} else {
	rayJob.Status.Reason = rayv1.SubmissionFailed
	rayJob.Status.Message = fmt.Sprintf("Job submission has failed. Reason: %s. Message: %s", cond.Reason, cond.Message)
	}
	return true
	}
	}
	return false
	}

Or to update RayJob status when failed to retrieve ray job?

kuberay/ray-operator/controllers/ray/rayjob_controller.go

Lines 271 to 284 in ea0b9c5

	jobInfo, err := rayDashboardClient.GetJobInfo(ctx, rayJobInstance.Status.JobId)
	if err != nil {
	// If the Ray job was not found, GetJobInfo returns a BadRequest error.
	if rayJobInstance.Spec.SubmissionMode == rayv1.HTTPMode && errors.IsBadRequest(err) {
	logger.Info("The Ray job was not found. Submit a Ray job via an HTTP request.", "JobId", rayJobInstance.Status.JobId)
	if _, err := rayDashboardClient.SubmitJob(ctx, rayJobInstance); err != nil {
	logger.Error(err, "Failed to submit the Ray job", "JobId", rayJobInstance.Status.JobId)
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, err
	}
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, nil
	}
	logger.Error(err, "Failed to get job info", "JobId", rayJobInstance.Status.JobId)
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, err
	}

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[anxietymonger]

@andrewsykim @kevin85421

[andrewsykim]

During the process, the submitter pod somehow goes into Complete state

@anxietymonger could you share logs from the submitter pod? I think it's important to identify why it went to completed state even when there's no running Ray cluster

[anxietymonger]

Thanks for the reply. Here is an example of the submitter pod.

2025-08-05 06:56:42,756	INFO cli.py:39 -- Job submission server address: http://a-4b5f-bcd1-122b571987f2-raycluster-cvjrq-head-svc.namespace.svc.cluster.local:8265
2025-08-05 06:56:43,776	SUCC cli.py:63 -- -----------------------------------------------------------------------
2025-08-05 06:56:43,777	SUCC cli.py:64 -- Job 'bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-j2gw7' submitted successfully
2025-08-05 06:56:43,777	SUCC cli.py:65 -- -----------------------------------------------------------------------
2025-08-05 06:56:43,777	INFO cli.py:289 -- Next steps
2025-08-05 06:56:43,777	INFO cli.py:290 -- Query the logs of the job:
2025-08-05 06:56:43,777	INFO cli.py:292 -- ray job logs bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-j2gw7
2025-08-05 06:56:43,777	INFO cli.py:294 -- Query the status of the job:
2025-08-05 06:56:43,777	INFO cli.py:296 -- ray job status bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-j2gw7
2025-08-05 06:56:43,777	INFO cli.py:298 -- Request the job to be stopped:
2025-08-05 06:56:43,777	INFO cli.py:300 -- ray job stop bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-j2gw7
2025-08-05 06:56:48,450	INFO cli.py:39 -- Job submission server address: http://a-4b5f-bcd1-122b571987f2-raycluster-cvjrq-head-svc.namespace.svc.cluster.local:8265
2025-08-05 06:56:43,654	INFO job_manager.py:531 -- Runtime env is setting up.
2025-08-05 06:56:53,308	INFO worker.py:1520 -- Using address 10.233.127.224:6379 set in the environment variable RAY_ADDRESS
2025-08-05 06:56:53,308	INFO worker.py:1660 -- Connecting to existing Ray cluster at address: 10.233.127.224:6379...
2025-08-05 06:56:53,320	INFO worker.py:1843 -- Connected to Ray cluster. View the dashboard at http://10.233.127.224:8265
2025-08-05 07:18:29,287	INFO cli.py:89 -- Status for job 'bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-j2gw7': RUNNING
2025-08-05 07:18:29,287	INFO cli.py:91 -- Status message: Job is currently running.

NAME                                                            READY   STATUS      RESTARTS   AGE
bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-qdskm                      0/1     Completed   0          86m
bdfa8ce3-4f3a-4b5f-bcd1-122b571987f2-raycluster-cv-head-rhjc9   1/1     Running     0          64m

The head pod was evicted at around 2025-08-05 07:18:00 (can't tell at which second).

[anxietymonger]

@anxietymonger could you share logs from the submitter pod? I think it's important to identify why it went to completed state even when there's no running Ray cluster

I'm not 100% sure how it works, but maybe because there IS a running cluster (but with no actual running job). I presume what's going on here is: after the head pod was evicted, the operator reconciles a new one, so the submitter pod was able to communicate with the head. The submitter completes since there is no log to follow. However, the part I can't understand is why the submitter could get the job's state even if the job no long exists.

I don't know the specification, but what is expected here is that the operater should:

• when the backofflimit is reached, release the raycluster and fail the rayjob
• when the backofflimit is not reached, resubmit the job

[anxietymonger]

@andrewsykim

[zshuying-field-ai]

I have faced the exact same issues!!

[zshuying-field-ai]

This significantly blocks my team from using RayJob. I have been wondering how there is no existing issues

[kevin85421]

Thanks @anxietymonger @zshuying-field-ai for reporting this issue.

cc @rueian @Future-Outlier can one of you take this issue or find a contributor to solve this ASAP?

Currently, JobDeploymentStatus will transition from JobDeploymentStatusRunning to terminal states (JobDeploymentStatusComplete or JobDeploymentStatusFailed) when both K8s Job submitter and the Ray job status are in terminal states. See the following code for more details.

kuberay/ray-operator/controllers/ray/rayjob_controller.go

Lines 268 to 300 in 97425fa

	if err != nil {
	// If the Ray job was not found, GetJobInfo returns a BadRequest error.
	if rayJobInstance.Spec.SubmissionMode == rayv1.HTTPMode && errors.IsBadRequest(err) {
	logger.Info("The Ray job was not found. Submit a Ray job via an HTTP request.", "JobId", rayJobInstance.Status.JobId)
	if _, err := rayDashboardClient.SubmitJob(ctx, rayJobInstance); err != nil {
	logger.Error(err, "Failed to submit the Ray job", "JobId", rayJobInstance.Status.JobId)
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, err
	}
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, nil
	}
	logger.Error(err, "Failed to get job info", "JobId", rayJobInstance.Status.JobId)
	return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, err
	}
	// If the JobStatus is in a terminal status, such as SUCCEEDED, FAILED, or STOPPED, it is impossible for the Ray job
	// to transition to any other. Additionally, RayJob does not currently support retries. Hence, we can mark the RayJob
	// as "Complete" or "Failed" to avoid unnecessary reconciliation.
	jobDeploymentStatus := rayv1.JobDeploymentStatusRunning
	reason := rayv1.JobFailedReason("")
	isJobTerminal := rayv1.IsJobTerminal(jobInfo.JobStatus)
	// If in K8sJobMode, further refine the terminal condition by checking if the submitter Job has finished.
	// See https://github.com/ray-project/kuberay/pull/1919 for reasons.
	if utils.HasSubmitter(rayJobInstance) {
	isJobTerminal = isJobTerminal && isSubmitterFinished
	}
	if isJobTerminal {
	jobDeploymentStatus = rayv1.JobDeploymentStatusComplete
	if jobInfo.JobStatus == rayv1.JobStatusFailed {
	jobDeploymentStatus = rayv1.JobDeploymentStatusFailed
	reason = rayv1.AppFailed
	}
	}

However, in this case, the submitter is in a terminal state, but the Ray job is not found because the head Pod fails and restarts. We should change the logic for the state transition to:

• JobDeploymentStatus will transition from JobDeploymentStatusRunning to terminal states (JobDeploymentStatusComplete or JobDeploymentStatusFailed) when: (1) both K8s Job submitter and the Ray job status are in terminal states or (2) K8s Job submitter is in terminal state and Ray job is not found.

Some ideas other than “Ray job is not found” (Idea 1) in (2):

• Idea 2: we can also force the JobDeploymentStatusFailed transition if it doesn’t occur within a certain period (e.g., 30s).
• Idea 3: We can implicitly set the head Pod’s restartPolicy to Never to avoid RayCluster deleting and recreating the head Pod, but this would be a bigger change. It’s not worth it for now.

Maybe we can implement both Idea 1 and Idea 2. Each one can resolve this issue.