Skip to content

Conversation

@guanguxiansheng
Copy link

@guanguxiansheng guanguxiansheng commented Dec 13, 2025

Motivation

We enabled the finalizer feature in our production environment (20,000 workflows), but encountered two online issues: 1. When a pending pod is deleted by another service, it gets stuck in the Terminating state, waiting for the finalizer to be cleaned up., however, the Pod DeleteFunc doesn't receive the deletion event; in fact, the UpdateFunc receives the update event, causing the Pod Finalizer to fail to clean up. 2. When the workflow controller restarts, the Pod AddFunc receives historical creation events and cannot clean up the finalizer.

Modifications

Regardless of the event detected in the Pod, the system will determine whether the Pod has been deleted based on the Pod DeletionTimestamp field and delete the Finalizer accordingly.
deleting := pod.DeletionTimestamp != nil c.commonPodEvent(ctx, pod, deleting)

Doucument

Like this, when pod is cleaned up by other services, DeletionTimestamp is set, this is the Update event.

@guanguxiansheng
Copy link
Author

@shuangkun hi, could you please help review this PR?

@MasonM
Copy link
Member

MasonM commented Dec 15, 2025

It looks like the DCO check failed. You should be able to fix that with these commands:

git commit --signoff --amend
git push --force

@Joibel Joibel self-assigned this Dec 15, 2025
@Joibel Joibel added the area/controller Controller issues, panics label Dec 15, 2025
@guanguxiansheng guanguxiansheng force-pushed the feature/remove-pod-finalizer branch from b744758 to b76944b Compare December 15, 2025 15:34
@guanguxiansheng
Copy link
Author

@MasonM thank you, I have already submit the commit, pls check again :)

@Joibel
Copy link
Member

Joibel commented Dec 16, 2025

/coderabbitai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller Controller issues, panics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants