APP-10775: viam training-script test-local #5524

vpandiarajan20 · 2025-11-21T20:07:29Z

Adds training-script test-local CLI command for testing ML training scripts locally in Docker before submitting to cloud.

Validates training script structure (setup.py, model/training.py) and dataset paths
Mounts local directories into Docker containers with proper working directory setup
Supports custom training arguments and both standard/custom container versions
Handles Docker availability checks, signal interrupts, and platform compatibility (linux/x86_64)
Added --force-linux-path flag to dataset export command to force Linux-style paths in dataset.jsonl - needed to run on Windows, so we have Linux paths (to use in the container)

Working on manual testing.

Apple Silicon Mac
Ubuntu
Intel Mac
Windows

Usage Example

Download the dataset

viam dataset export --destination=tmp/tmp --dataset-id=6515b637a420eb685e988e84

Clone the classification-tflite repo

Run the below command

viam training-script test-local \
  --dataset-root=/Users/[email protected]/viam/rdk \    
  --dataset-file=tmp/tmp/dataset.jsonl \
  --training-script-directory=/Users/[email protected]/viam-modules/classification-tflite \
  --container-version=tf:2.16 \                                                                                                                       
  --model-output-directory=./output \
  --custom-args=num_epochs=1

cli/ml_training.go

vpandiarajan20 · 2025-11-21T20:43:56Z

cli/ml_training.go

+		// Validate that the key portion only contains safe characters
+		parts := strings.SplitN(arg, "=", 2)
+		key := parts[0]
+		if !isValidArgumentKey(key) {


I tried to clean the input, let me know if there's a better way to do this.

vpandiarajan20 · 2025-11-21T20:46:55Z

cli/ml_training.go

+
+		// Provide additional context for platform-related errors
+		errMsg := err.Error()
+		if strings.Contains(errMsg, "platform") || strings.Contains(errMsg, "architecture") {


this is kind of ugly, but I thought it was worth including. it's also in the command description so potentially removable.

cli/app.go

vpandiarajan20 · 2025-11-24T23:01:32Z

cli/app.go

+					Flags: []cli.Flag{
+						&cli.StringFlag{
+							Name:     trainFlagDatasetRoot,
+							Usage:    "path to the dataset root directory (where dataset.jsonl and image files are located). This is where you ran the 'viam dataset export' command from.",


Let me know if this is too confusing.

vpandiarajan20 · 2025-11-24T23:02:31Z

cli/app.go

+          └── images/
+              └── cat.jpg
+
+NOTES:


Maybe this goes in documentation? I also want to add that if the containers really slow, it could be because the dataset root or training script directory has a bunch of extra files. Apparently mounting volumes can be expensive.

etai-shuchatowitz · 2025-12-04T18:31:24Z

cli/ml_training.go

+	cmd.Stderr = c.App.ErrWriter
+
+	printf(c.App.Writer, "WARNING: If this is your first time running training, "+
+		"it may take a few minutes to download the container image. "+


A few minutes is a little bit of an understatement :)

njooma

Looks good from SDK perspective, did not review implementation details

allisonschiang

LGTM

etai-shuchatowitz

LGTM!

first pass

fdff64b

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Nov 21, 2025

security and linting

fb04bd4

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 21, 2025

descrip text

214911d

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 21, 2025

vpandiarajan20 commented Nov 21, 2025

View reviewed changes

cli/ml_training.go Outdated Show resolved Hide resolved

vpandiarajan20 commented Nov 21, 2025

View reviewed changes

vpandiarajan20 requested a review from etai-shuchatowitz November 24, 2025 18:08

Merge branch 'viamrobotics:main' into APP-10775

9c22fec

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 24, 2025

dataset paths

c76a430

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 24, 2025

CLI help text

a8f8f8d

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 24, 2025

lint

3f4da57

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 24, 2025

vpandiarajan20 commented Nov 24, 2025

View reviewed changes

clean up

e28a0d1

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 25, 2025

Merge branch 'main' into APP-10775

753882e

viambot removed the safe to test This pull request is marked safe to test from a trusted zone label Nov 25, 2025

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 1, 2025

vpandiarajan20 added 2 commits December 1, 2025 16:10

warning message

704f696

warning message 2

1fad4ba

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 1, 2025

change help text

001c886

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 2, 2025

vpandiarajan20 requested a review from etai-shuchatowitz December 4, 2025 18:21

etai-shuchatowitz reviewed Dec 4, 2025

View reviewed changes

remove validation

994b80e

viambot removed the safe to test This pull request is marked safe to test from a trusted zone label Dec 4, 2025

lint

c9d36d7

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 4, 2025

to make windows work

b6efdeb

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 5, 2025

Merge branch 'viamrobotics:main' into APP-10775

916fa04

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 5, 2025

njooma approved these changes Dec 5, 2025

View reviewed changes

allisonschiang reviewed Dec 5, 2025

View reviewed changes

help text

4bc5312

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 5, 2025

etai-shuchatowitz approved these changes Dec 8, 2025

View reviewed changes

vpandiarajan20 merged commit 909e878 into viamrobotics:main Dec 10, 2025
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

APP-10775: viam training-script test-local #5524

APP-10775: viam training-script test-local #5524

Uh oh!

vpandiarajan20 commented Nov 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

vpandiarajan20 Nov 21, 2025

Uh oh!

vpandiarajan20 Nov 21, 2025

Uh oh!

Uh oh!

vpandiarajan20 Nov 24, 2025

Uh oh!

vpandiarajan20 Nov 24, 2025

Uh oh!

etai-shuchatowitz Dec 4, 2025

Uh oh!

njooma left a comment

Uh oh!

allisonschiang left a comment

Uh oh!

etai-shuchatowitz left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

APP-10775: viam training-script test-local #5524

APP-10775: viam training-script test-local #5524

Uh oh!

Conversation

vpandiarajan20 commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage Example

Uh oh!

Uh oh!

vpandiarajan20 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

vpandiarajan20 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vpandiarajan20 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

vpandiarajan20 Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

etai-shuchatowitz Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

njooma left a comment

Choose a reason for hiding this comment

Uh oh!

allisonschiang left a comment

Choose a reason for hiding this comment

Uh oh!

etai-shuchatowitz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vpandiarajan20 commented Nov 21, 2025 •

edited

Loading