Consider the way to inform users that the specified gang scheduler isn't supported

[tenzen-y]

Since #1747, we have treated the scheduler-plugins as a default gang-scheduler, and we use the scheduler-plugins if users set anything other than volcano to https://github.com/kubeflow/training-operator/blob/6d6d83dfe7d8c3f4f3938c00c04d8b7410caa205/cmd/training-operator.v1/main.go#LL76C1-L77C105.

So, if the specified gang scheduler isn't supported in the training operator, informing that would be better.

/cc @johnugeorge

[tenzen-y]

/remove-kind discussion
/kind feature

[tenzen-y]

cc: @Syulin7

[Syulin7]

Do we need to check if the podgroup CRD exists? Users should ensure that their scheduler can support Volcano podgroup or scheduler-plugins podgroup.

[tenzen-y]

Do we need to check if the podgroup CRD exists? Users should ensure that their scheduler can support Volcano podgroup or scheduler-plugins podgroup.

It sounds good to me. If users set --gang-scheduler-name=volcano, the training operator checks whether there is the podgroup CRD in the cluster; if users set another value to --gang-scheduler-name, the training operator checks the scheduler-plugins CRD in the cluster.

@johnugeorge wdyt?

[johnugeorge]

The key question is, Are we planning to support only volcano and scheduler plugins in the future? If we are,(due to overhead in management), it sounds good to me. Else, it might be better to make CRD check to be dynamic(like in https://github.com/kubeflow/katib/blob/master/cmd/katib-controller/v1beta1/main.go#L62)

[tenzen-y]

Else, it might be better to make CRD check to be dynamic(like in https://github.com/kubeflow/katib/blob/master/cmd/katib-controller/v1beta1/main.go#L62)

Makes sense.
We probably need to support native PodGroup CRD (kube-controller-manager) once the feature graduates to beta.

[tenzen-y]

@Syulin7 Any thoughts?

[Syulin7]

Else, it might be better to make CRD check to be dynamic(like in https://github.com/kubeflow/katib/blob/master/cmd/katib-controller/v1beta1/main.go#L62)

Agree, users decide whether or not to check the podgroup CRD themselves?

[tenzen-y]

Else, it might be better to make CRD check to be dynamic(like in https://github.com/kubeflow/katib/blob/master/cmd/katib-controller/v1beta1/main.go#L62)

Agree, users decide whether or not to check the podgroup CRD themselves?

Probably, we can have podgroups for the volcano and the scheudler-plugins as a default.

[tenzen-y]

@Syulin7 Do you have enough time to work on this?

[Syulin7]

/assign