Delete Kubernetes resources when the client waits for and sees app completion

[liyinan926]

What changes were proposed in this pull request?

This PR fixes #519 for the case where the submission client waits for the submitted application to finish. Upon completion of the application, the submission client deletes all Kubernetes resources created for the application to run.

[mccheah]

One other caveat is that we lose some state that might be useful for debugging the driver. For example, we lose the ConfigMap object that is required to set up the init-container, which is useful for knowing what the properties resolved to if the init-container fails. I'm not sure how problematic this will be in practice, but it is something to consider.

[liyinan926]

The ConfigMap is purely an implementation detail and really shouldn't be exposed to the users. For the purpose of debugging, I think it's better logging the ConfigMap contents instead of leaving the ConfigMap around after an application finished.

[felixcheung]

ConfigMap aside, I think the notion of keeping pod for debugging make sense, I thought we had a config on that (it might be a different area/code path)

[liyinan926]

rerun integration tests please

[liyinan926]

I want to re-iterate on this issue/PR. If we have concern around losing some objects like the ConfigMap for setting up the init-container, as I said above, we could log information stored in it for debugging purpose. This, IMO, is better than making the ConfigMap stick around just for debugging. Thoughts?

@mccheah @foxish

[dharmeshkakadia]

Any thoughts on this ? It would be good to cleanup resources after completion. In a normal scenario, this is filling up a lot of services in completed state for example.

[felixcheung]

hey this sounds useful to have

[felixcheung]

ConfigMap aside, I think the notion of keeping pod for debugging make sense, I thought we had a config on that (it might be a different area/code path)

[felixcheung]

Warn if it will be GC'd?

[liyinan926]

Done.

[dharmeshkakadia]

Thanks @felixcheung for jumping on this :)

[felixcheung]

hey where are we on this?
and how about going upstream?

[liyinan926]

@felixcheung Yes, I think we should go upstream. I created https://issues.apache.org/jira/browse/SPARK-23571. Also given that we are in the process of getting rid of the init-container, the ConfigMap for the init-container will be gone also. So it makes more sense to clean up after application completion.

[foxish]

Sorry, didn't see this before. Same comment as in apache#20722 (comment). Why not do this during driver.stop()? - that way, 1) if we lose the driver, k8s garbage collection cleans up everything 2) if driver terminates, we clean up executors as well as auxiliary resources like configmaps etc.

[foxish]

I want to re-iterate on this issue/PR. If we have concern around losing some objects like the ConfigMap for setting up the init-container, as I said above, we could log information stored in it for debugging purpose. This, IMO, is better than making the ConfigMap stick around just for debugging. Thoughts?

I agree. We can dump all k8s objects. My hunch is that it's not that useful, given it's a pretty deeply buried implementation detail.

[liyinan926]

As discussed in apache#20722, we think the right solution is move resource management into the driver pod. This way, cleanup of auxiliary resources upon completion is guaranteed regardless of which deployment mode is used and whether the client waits for application to complete or not.