Skip to content

update the logic for updating the k8s resourcces #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ibm-ci-bot merged 3 commits into from
Sep 30, 2021
Merged

update the logic for updating the k8s resourcces #752

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8a187a0
update the logic for updating the k8s resourcces
Daniel-Fan Sep 30, 2021
5491916
wait the deletion of existing k8s resource
Daniel-Fan Sep 30, 2021
657dd16
remove debug info
Daniel-Fan Sep 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
wait the deletion of existing k8s resource
  • Loading branch information
@Daniel-Fan
Daniel-Fan committed Sep 30, 2021
commit 54919165f00b363159234cbd5ea54d1f6a4b16fb
75 changes: 40 additions & 35 deletions controllers/operandrequest/reconcile_operand.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -797,6 +797,9 @@ func (r *Reconciler) createK8sResource(ctx context.Context, k8sResTemplate unstr
if err != nil && !apierrors.IsAlreadyExists(err) {
return errors.Wrap(err, "failed to create k8s resource")
}
if apierrors.IsAlreadyExists(err) {
klog.Infof("resource already exists -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

klog.V(2).Infof("Finish creating the k8s Resource: -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)

Expand All @@ -809,52 +812,54 @@ func (r *Reconciler) updateK8sResource(ctx context.Context, existingK8sRes unstr
name := existingK8sRes.GetName()
namespace := existingK8sRes.GetNamespace()

// Update the k8s res
err := wait.PollImmediate(constant.DefaultCRFetchPeriod, constant.DefaultCRFetchTimeout, func() (bool, error) {
existingK8sRes = unstructured.Unstructured{
Object: map[string]interface{}{
"apiVersion": apiversion,
"kind": kind,
},
}

existingK8sRes := unstructured.Unstructured{
Object: map[string]interface{}{
"apiVersion": apiversion,
"kind": kind,
},
}
if err := r.Client.Get(ctx, types.NamespacedName{
Name: name,
Namespace: namespace,
}, &existingK8sRes); err != nil {
return errors.Wrapf(err, "failed to get k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

if !checkLabel(existingK8sRes, map[string]string{constant.OpreqLabel: "true"}) {
return nil
}

// delete the existing k8s resource, some resources could not be updated after it has been created
if err := r.Client.Delete(ctx, &existingK8sRes); err != nil {
Copy link
Contributor

@horis233 horis233 Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this line cause the k8s resource to get deleted in every reconcile loop?

Copy link
Contributor Author

@Daniel-Fan Daniel-Fan Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, when ODLM is reconciling, and the resources is forced to update and label is correct. It will be deleted and re-created

Copy link
Contributor Author

@Daniel-Fan Daniel-Fan Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But even if we use the Update(), the resource will be updated in each reconcile loop except the resource could not be updated.

One solution is to compare the existing resource with the new template, and decide whether we should update it.

Copy link
Contributor

@horis233 horis233 Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I ask which k8s resouce, that can't be updated?

Copy link
Contributor Author

@Daniel-Fan Daniel-Fan Sep 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like job, some of fields in template is immutable, so we could not update it.

return errors.Wrapf(err, "failed to delete existing k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

// wait for the resource has been deleted
err := wait.PollImmediate(constant.DefaultCRFetchPeriod, constant.DefaultCRFetchTimeout, func() (bool, error) {
err := r.Client.Get(ctx, types.NamespacedName{
Name: name,
Namespace: namespace,
}, &existingK8sRes)

if err != nil {
return false, errors.Wrapf(err, "failed to get k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

if !checkLabel(existingK8sRes, map[string]string{constant.OpreqLabel: "true"}) {
if err == nil {
// continue wait for the deletion
return false, nil
} else if err != nil && apierrors.IsNotFound(err) {
return true, nil
}

// delete the existing k8s resource, some resources could not be updated after it has been created
err = r.Client.Delete(ctx, &existingK8sRes)

if err != nil {
return false, errors.Wrapf(err, "failed to delete existing k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

var newK8sRes unstructured.Unstructured
newK8sRes.SetAPIVersion(apiversion)
newK8sRes.SetKind(kind)
newK8sRes.SetName(name)
newK8sRes.SetNamespace(namespace)

err = r.createK8sResource(ctx, newK8sRes, k8sResConfig, newLabels, newAnnotations)

if err != nil {
return false, errors.Wrapf(err, "failed to update k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

return true, nil
return false, errors.Wrapf(err, "failed to wait k8s resource deleted -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
})

if err != nil {
return errors.Wrapf(err, "failed to wait k8s resource deleted -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

var k8sRes unstructured.Unstructured
k8sRes.SetAPIVersion(apiversion)
k8sRes.SetKind(kind)
k8sRes.SetName(name)
k8sRes.SetNamespace(namespace)
if err = r.createK8sResource(ctx, k8sRes, k8sResConfig, newLabels, newAnnotations); err != nil {
return errors.Wrapf(err, "failed to update k8s resource -- Kind: %s, NamespacedName: %s/%s", kind, namespace, name)
}

Expand Down