Skip to content

Adding Training image needed for train api #1963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
google-oss-prow merged 14 commits into from
Jan 11, 2024
Merged

Adding Training image needed for train api #1963

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a2fc0cb
adding training image creationcode
deepanker13 Dec 12, 2023
3197a94
reformatting code using black
deepanker13 Dec 12, 2023
5c2d48c
code review changes
deepanker13 Dec 13, 2023
c7260da
fixes
deepanker13 Dec 20, 2023
61fb7e1
running workflow on push
deepanker13 Dec 21, 2023
94d0543
correcting context
deepanker13 Dec 21, 2023
5931a49
correcting context
deepanker13 Dec 21, 2023
cf6419e
fixing trainer code
deepanker13 Jan 4, 2024
3204149
code review changes
deepanker13 Jan 8, 2024
4642b9e
resolving merge conflict
deepanker13 Jan 10, 2024
112c581
code review changes
deepanker13 Jan 11, 2024
1034403
downgrading pytorch version, removing changes for running things loca…
deepanker13 Jan 11, 2024
f520329
ci fix and removing unused parameter and adding check if pvc exists a…
deepanker13 Jan 11, 2024
95b3e2b
gpu training example fix
deepanker13 Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
code review changes
  • Loading branch information
@deepanker13
deepanker13 committed Jan 10, 2024
commit 32041490e003a5e13d686fd70392173d2ce1fc20
4 changes: 3 additions & 1 deletion .github/workflows/publish-example-images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ jobs:
- component-name: mxnet-auto-tuning
dockerfile: examples/mxnet/tune/Dockerfile
context: examples/mxnet/tune

- component-name: train-api-hf-image
dockerfile: sdk/python/kubeflow/trainer/hf_dockerfile
context: sdk/python/kubeflow/trainer
# TODO (tenzen-y): Fix the below broken Dockerfiles
# - component-name: pytorch-dist-mnist-mpi
# dockerfile: examples/pytorch/mnist/Dockerfile-mpi
Expand Down
26 changes: 0 additions & 26 deletions .github/workflows/publish-sdk-images.yaml
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion sdk/python/kubeflow/trainer/hf_dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Use an official Pytorch runtime as a parent image
FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
FROM nvcr.io/nvidia/pytorch:23.12-py3
Copy link
Member

@andreyvelich andreyvelich Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to use PyTorch image from NVIDIA for this trainer ?
Would it be better to take official PyTorch image similar to what we use in SDK ?
docker.io/pytorch/pytorch:1.12.1-cuda11.3-cudnn8-runtime

Copy link
Contributor Author

@deepanker13 deepanker13 Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as suggested by @tenzen-y
#1963 (comment)

Copy link
Member

@andreyvelich andreyvelich Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. @tenzen-y Do you know if PyTorch has any official image that we can use that is supported on all platforms ?

Copy link
Member

@tenzen-y tenzen-y Jan 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreyvelich As I remember correctly, the PyTorch doesn't provide images with multiple architecture platforms with GPU. So, we need to use the NVIDIA official images.


# Set the working directory in the container
WORKDIR /app
Expand Down