Skip to content

fix(compose): forward the autocast dtype overrides the runtime reads - #594

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:feat-compose-amp-env
Sep 27, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:feat-compose-amp-env

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

The gap

laya/agent.py picks the autocast dtype a forward runs in from the environment — LAYA_CUDA_AMP
at laya/agent.py:211, inside _cuda_amp_dtype, and LAYA_CPU_AMP at laya/agent.py:456, in the
CPU branch of Agent.__init__. The Compose deployment that selects the device never mentions either
name, so the published images serve the checkpoint's own amp_dtype no matter what the operator
exports.
The deployment that chooses where the model runs drops the knob that chooses in what precision it
runs.

Before / after

Measured on this machine, Docker 29.6.2 / Compose v5.3.1, same command both times:

### BEFORE (main, 4066d5d)
$ LAYA_CUDA_AMP=fp16 docker compose -f compose.yaml -f compose.cuda.yaml config | grep LAYA_
      LAYA_DEVICE: cuda
      LAYA_MODEL: auto
      LAYA_MODEL_PATH: ""
      LAYA_REQUEST_FILE: /opt/laya/examples/request.json
      LAYA_DEVICE: cuda

### AFTER (this branch)
      LAYA_CPU_AMP: ""
      LAYA_CUDA_AMP: fp16
      LAYA_DEVICE: cuda
      LAYA_MODEL: auto
      LAYA_MODEL_PATH: ""
      LAYA_REQUEST_FILE: /opt/laya/examples/request.json
      LAYA_CPU_AMP: ""
      LAYA_CUDA_AMP: fp16
      LAYA_DEVICE: cuda

The variable was set, exported, and visible to docker compose config in both cases. The only
thing that changed is whether it survives the boundary into the container's environment.

And on the CPU-only stack, which has no GPU file to pass it through:

$ LAYA_CPU_AMP=bf16 docker compose -f compose.yaml -f compose.http.yaml config | grep -E "LAYA_(CPU|CUDA)_AMP"
      LAYA_CPU_AMP: bf16      LAYA_CUDA_AMP: ""      # laya
      LAYA_CPU_AMP: bf16      LAYA_CUDA_AMP: ""      # laya-serve

Why this is a decision, not a rounding difference

Quoting the repo's own measurement rather than mine — README's "Automated Confidence Gating"
section, on the fixed set from benchmarks/parity_fast.py (60 states, 288 questions per
checkpoint, RTX 2000 Ada): bf16 moves a probability by up to 0.073 against the fp32 forward and
flips 3 of 864 argmaxes across the three checkpoints, where fp16 stays within 0.019 and
flips none, at the same latency. The same paragraph tells the reader to "fit and measure a
threshold in the dtype you serve with" — which the Compose quickstart currently cannot do, because
it cannot select the dtype. So the empty default here is not a change of behaviour: unset renders
as empty, empty is ignored by _cuda_amp_dtype, and the image keeps serving the checkpoint's
amp_dtype exactly as it does today.

Verified on this box against real weights (CPU build): _cuda_amp_dtype returns torch.float16
for fp16, torch.bfloat16 for bf16, and the checkpoint default for unset, "", and fp99.
What I could not exercise here is the CUDA forward itself — no NVIDIA GPU on this machine — so the
dtype selection is measured and the GPU throughput claim is README's, not mine.

What changed

  • compose.yaml, compose.cuda.yaml (both services), compose.spark.yaml (both services),
    compose.http.yaml: forward LAYA_CUDA_AMP and LAYA_CPU_AMP as ${VAR:-} host passthroughs.
    Both GPU services need their own copy — the file's existing comment already says an override on
    laya never reaches laya-serve.
  • docs/docker.md: one Configuration-table row per name, next to LAYA_DEVICE; and the paragraph
    that already explained the HF_HOME exception now names this one too.

What is deliberately not forwarded

LAYA_MPS_AMP_MIN_ROWS is the third dtype name in laya/agent.py, and I forwarded it first and
then took it back out. docs/docker.md already says Apple MPS containers are outside the
quickstart, and no compose file here can select mps — a knob that reaches nothing would look
like thoroughness and read as noise. So the exclusion is stated on the page, and the gate checks
that the page still gives the reason rather than freezing the reason in a code comment.

The gate

tests/test_compose_env.py — weight-free, torch-free, no third-party YAML, same textual style as
the Compose section of tests/test_packaging.py. It passes with 51 checks, 0 failures.

The names are derived, not transcribed: it reads laya/agent.py and takes every
os.environ.get("LAYA_…") that sits within four lines of a dtype-decision site
(torch.float16|bfloat16|float32 or amp_dtype). Prose that merely names a precision is not a
decision site, so LAYA_MPS_AMP_MIN_ROWS shows up in the excluded set instead of the required
one. The drift that hid these two names in the first place was three-way — package, markdown,
Compose — and a literal list in the test would go stale the same way. Then, in both directions:

  1. the derived set is exactly the two dtype names, and each is still read by the core (so the gate
    cannot pass vacuously by parsing nothing);
  2. every compose file that pins LAYA_DEVICE forwards every derived name, in every service it
    defines, and in the ${NAME:-…} form — a hardcoded "fp16" would pin a dtype for every user,
    and a bare ${NAME} with no default would make a host that never mentions it ask for one;
  3. an empty default is only safe because the core reads it as os.environ.get(name, "") — checked
    out of the core, not asserted from the docs;
  4. docs/docker.md has a table row per name;
  5. the MPS exclusion stays falsifiable: the excluded set is exactly [LAYA_MPS_AMP_MIN_ROWS], no
    compose file defaults to a device outside {cpu, cuda}, and the docs sentence still carries the
    reason;
  6. when docker is installed, render compose.yaml + compose.cuda.yaml with LAYA_CUDA_AMP=fp16
    and require the value in both GPU services. No docker → a NOTE, never a pass.

Not vacuous: the same gate file run against unmodified main gives 16 named failures and
exit 1
— every service in every file, both docs rows, the MPS reason, and the rendered
environment (FAIL render/both GPU services get LAYA_CUDA_AMP=fp16: {'laya': None, 'laya-serve': None}).

Mutation-checked, 11 variants, each breaking one property the gate decides: 11/11 caught,
tree restored byte-identical, exit 0.
A service loses a name; a hardcoded fp16; a whole file
missing both; a bare ${VAR}; the CUDA docs row deleted; the MPS sentence deleted; the MPS name
kept but its reason deleted; the core renamed its lookup; the dtype-site regex made non-matching so
the derived set empties; a compose file starting to default to mps; and one that matters more
than it looks —

m10 a GPU service hides behind a profile: only the renderer sees it
    exit=1  FAIL render/both GPU services get LAYA_CUDA_AMP=fp16: {'laya-serve': 'fp16'}

That last one leaves every textual check green, because the environment: block is untouched in
the file; the service simply stops being part of a default render. It is the reason check 6 is not
redundant with checks 2 and 3.

CI

One line each in ci.yml and release.yml's weight-free lists, after tests/test_packaging.py —
the same shape as #575's test_env_docs.py. Locally: ruff check laya/ --select=E9,F63,F7,F82, F401,F811 --line-length=120 clean, compileall -q laya/ tests/ clean, tests/test_packaging.py
43 passed, tests/test_compose_env.py 51 passed, zensical build --strict --clean → "No issues
found", docker compose config rendering as quoted above.

Overlap, checked before writing code

Swept the file lists of all 151 open PRs. compose.yaml, compose.cuda.yaml,
compose.spark.yaml, tests/test_compose_env.py and laya/agent.py (read, not edited) are in no
open PR.

laya/agent.py selects the forward's dtype from LAYA_CUDA_AMP and LAYA_CPU_AMP, and no
compose file that sets LAYA_DEVICE passed either name, so the published deployment always
served the checkpoint's amp_dtype whatever the operator exported. README measures the
difference as decisions, not rounding: bf16 flips 3 of 864 argmaxes on the parity_fast set
where fp16 flips none, at the same latency.

Both GPU services get the pair, because an override on `laya` never reaches `laya-serve`.
An empty value means "the checkpoint's own amp_dtype", which is what these images shipped
before, so the passthrough default changes nothing until it is set. The MPS row gate is
deliberately left out: no container here can select MPS.
…ployment

The names are derived from laya/agent.py by reading the env lookups that sit within a few
lines of a dtype decision, so the list cannot go stale when the core reads another one, and
a regex that stops matching is a failure rather than a silent pass. Each compose file that
pins LAYA_DEVICE must forward every derived name in every service block, as a host
passthrough rather than a hardcoded dtype, and docs/docker.md must carry a row per name.

The exclusion is falsifiable: MPS is not required because no compose file defaults to that
device, and the check asserts that. Where docker is installed the gate also renders
compose.yaml + compose.cuda.yaml and requires the value in both services' environments;
without docker it prints a note and the textual checks still decide the exit code.
…er pinned or made required

A bare ${LAYA_CUDA_AMP} is technically a passthrough but makes the variable
required: a deployment that never mentions it would be treated as asking for a
dtype, and the default the core implements (empty = the checkpoint's own
amp_dtype) would never be reached.
A docs row that names the variable without saying why is the state the gap was
found in: the reader cannot tell an intentional exclusion from an oversight.
@NandhaKishorM
NandhaKishorM merged commit c16b852 into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants