SDK sets log configuration globally in a way that negatively impacts other dependencies

[droctothorpe]

https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L27-L28
https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/utils/utils.py#L24-L25

These two lines in these two modules configure logging globally in a way that impacts other libraries. I'm using Dask Kubernetes which uses httpx; httpx does a ton of logging to info, which is fine, since Python defaults to a log level of warning, but if I import the training operator SDK, it overwrites the level to info globally, which causes httpx to print a ton of crud.

As far as I can tell, this is only being set in training_client.py and utils.py. Would you object to replacing these log statements with simple print statements, since that doesn't impact other dependencies? Happy to make the contribution.

[droctothorpe]

@andreyvelich

[johnugeorge]

@droctothorpe We can skip setting default to Info level. Current logs can be converted to debug logs. For pod logs, we can return as like in https://github.com/kubernetes-client/python/blob/master/kubernetes/client/api/logs_api.py#L131

/cc @tenzen-y

[droctothorpe]

@johnugeorge there are several calls to logging.info that convey information that is visible by default in the current pattern, e.g.:

• https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L225
• https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L225
• https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L225

I suspect that that's probably why the log level was set to info in the first place. Are you okay with disabling those logs unless the user enables debug logs? If not, we may want to replace those specific statements with calls to print (or do some complex stuff with dictConfig).

Happy to follow your lead, just wanted to convey the one reservation I had (i.e. those logging.info statements no longer being visible by default).

[johnugeorge]

Yes. To me, the current logs are extra info logs which needs to be enabled only in debug mode.

[droctothorpe]

Thanks for clarifying, @johnugeorge. I'll submit a PR.