Repository navigation
fix(compose): forward the autocast dtype overrides the runtime reads - #594
Merged
NandhaKishorM merged 4 commits intoSep 27, 2026
Merged
Conversation
laya/agent.py selects the forward's dtype from LAYA_CUDA_AMP and LAYA_CPU_AMP, and no compose file that sets LAYA_DEVICE passed either name, so the published deployment always served the checkpoint's amp_dtype whatever the operator exported. README measures the difference as decisions, not rounding: bf16 flips 3 of 864 argmaxes on the parity_fast set where fp16 flips none, at the same latency. Both GPU services get the pair, because an override on `laya` never reaches `laya-serve`. An empty value means "the checkpoint's own amp_dtype", which is what these images shipped before, so the passthrough default changes nothing until it is set. The MPS row gate is deliberately left out: no container here can select MPS.
…ployment The names are derived from laya/agent.py by reading the env lookups that sit within a few lines of a dtype decision, so the list cannot go stale when the core reads another one, and a regex that stops matching is a failure rather than a silent pass. Each compose file that pins LAYA_DEVICE must forward every derived name in every service block, as a host passthrough rather than a hardcoded dtype, and docs/docker.md must carry a row per name. The exclusion is falsifiable: MPS is not required because no compose file defaults to that device, and the check asserts that. Where docker is installed the gate also renders compose.yaml + compose.cuda.yaml and requires the value in both services' environments; without docker it prints a note and the textual checks still decide the exit code.
…er pinned or made required
A bare ${LAYA_CUDA_AMP} is technically a passthrough but makes the variable
required: a deployment that never mentions it would be treated as asking for a
dtype, and the default the core implements (empty = the checkpoint's own
amp_dtype) would never be reached.
A docs row that names the variable without saying why is the state the gap was found in: the reader cannot tell an intentional exclusion from an oversight.
Owner
|
Thanks @aashish254, merged. It ships in 0.3.21. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
laya/agent.pypicks the autocast dtype a forward runs in from the environment —LAYA_CUDA_AMPat
laya/agent.py:211, inside_cuda_amp_dtype, andLAYA_CPU_AMPatlaya/agent.py:456, in theCPU branch of
Agent.__init__. The Compose deployment that selects the device never mentions eithername, so the published images serve the checkpoint's own
amp_dtypeno matter what the operatorexports.
The deployment that chooses where the model runs drops the knob that chooses in what precision it
runs.
Before / after
Measured on this machine, Docker 29.6.2 / Compose v5.3.1, same command both times:
The variable was set, exported, and visible to
docker compose configin both cases. The onlything that changed is whether it survives the boundary into the container's environment.
And on the CPU-only stack, which has no GPU file to pass it through:
Why this is a decision, not a rounding difference
Quoting the repo's own measurement rather than mine — README's "Automated Confidence Gating"
section, on the fixed set from
benchmarks/parity_fast.py(60 states, 288 questions percheckpoint, RTX 2000 Ada): bf16 moves a probability by up to 0.073 against the fp32 forward and
flips 3 of 864 argmaxes across the three checkpoints, where fp16 stays within 0.019 and
flips none, at the same latency. The same paragraph tells the reader to "fit and measure a
threshold in the dtype you serve with" — which the Compose quickstart currently cannot do, because
it cannot select the dtype. So the empty default here is not a change of behaviour: unset renders
as empty, empty is ignored by
_cuda_amp_dtype, and the image keeps serving the checkpoint'samp_dtypeexactly as it does today.Verified on this box against real weights (CPU build):
_cuda_amp_dtypereturnstorch.float16for
fp16,torch.bfloat16forbf16, and the checkpoint default for unset,"", andfp99.What I could not exercise here is the CUDA forward itself — no NVIDIA GPU on this machine — so the
dtype selection is measured and the GPU throughput claim is README's, not mine.
What changed
compose.yaml,compose.cuda.yaml(both services),compose.spark.yaml(both services),compose.http.yaml: forwardLAYA_CUDA_AMPandLAYA_CPU_AMPas${VAR:-}host passthroughs.Both GPU services need their own copy — the file's existing comment already says an override on
layanever reacheslaya-serve.docs/docker.md: one Configuration-table row per name, next toLAYA_DEVICE; and the paragraphthat already explained the
HF_HOMEexception now names this one too.What is deliberately not forwarded
LAYA_MPS_AMP_MIN_ROWSis the third dtype name inlaya/agent.py, and I forwarded it first andthen took it back out.
docs/docker.mdalready says Apple MPS containers are outside thequickstart, and no compose file here can select
mps— a knob that reaches nothing would looklike thoroughness and read as noise. So the exclusion is stated on the page, and the gate checks
that the page still gives the reason rather than freezing the reason in a code comment.
The gate
tests/test_compose_env.py— weight-free, torch-free, no third-party YAML, same textual style asthe Compose section of
tests/test_packaging.py. It passes with 51 checks, 0 failures.The names are derived, not transcribed: it reads
laya/agent.pyand takes everyos.environ.get("LAYA_…")that sits within four lines of a dtype-decision site(
torch.float16|bfloat16|float32oramp_dtype). Prose that merely names a precision is not adecision site, so
LAYA_MPS_AMP_MIN_ROWSshows up in the excluded set instead of the requiredone. The drift that hid these two names in the first place was three-way — package, markdown,
Compose — and a literal list in the test would go stale the same way. Then, in both directions:
cannot pass vacuously by parsing nothing);
LAYA_DEVICEforwards every derived name, in every service itdefines, and in the
${NAME:-…}form — a hardcoded"fp16"would pin a dtype for every user,and a bare
${NAME}with no default would make a host that never mentions it ask for one;os.environ.get(name, "")— checkedout of the core, not asserted from the docs;
docs/docker.mdhas a table row per name;[LAYA_MPS_AMP_MIN_ROWS], nocompose file defaults to a device outside
{cpu, cuda}, and the docs sentence still carries thereason;
compose.yaml + compose.cuda.yamlwithLAYA_CUDA_AMP=fp16and require the value in both GPU services. No docker → a NOTE, never a pass.
Not vacuous: the same gate file run against unmodified
maingives 16 named failures andexit 1 — every service in every file, both docs rows, the MPS reason, and the rendered
environment (
FAIL render/both GPU services get LAYA_CUDA_AMP=fp16: {'laya': None, 'laya-serve': None}).Mutation-checked, 11 variants, each breaking one property the gate decides: 11/11 caught,
tree restored byte-identical, exit 0. A service loses a name; a hardcoded
fp16; a whole filemissing both; a bare
${VAR}; the CUDA docs row deleted; the MPS sentence deleted; the MPS namekept but its reason deleted; the core renamed its lookup; the dtype-site regex made non-matching so
the derived set empties; a compose file starting to default to
mps; and one that matters morethan it looks —
That last one leaves every textual check green, because the
environment:block is untouched inthe file; the service simply stops being part of a default render. It is the reason check 6 is not
redundant with checks 2 and 3.
CI
One line each in
ci.ymlandrelease.yml's weight-free lists, aftertests/test_packaging.py—the same shape as #575's
test_env_docs.py. Locally:ruff check laya/ --select=E9,F63,F7,F82, F401,F811 --line-length=120clean,compileall -q laya/ tests/clean,tests/test_packaging.py43 passed,
tests/test_compose_env.py51 passed,zensical build --strict --clean→ "No issuesfound",
docker compose configrendering as quoted above.Overlap, checked before writing code
Swept the file lists of all 151 open PRs.
compose.yaml,compose.cuda.yaml,compose.spark.yaml,tests/test_compose_env.pyandlaya/agent.py(read, not edited) are in noopen PR.
compose.http.yaml: docs(security): document opt-in revision pinning and digest verification #528 insertsLAYA_SHA256_DIGESTSafterLAYA_LOG_LEVEL(
@@ -42,6 +42,7 @@), four lines below my insert afterLAYA_DEVICE. Our hunks share context, sowhichever lands second will probably need a rebase — keep both lines, nothing about them
conflicts semantically. feat(serve): add a /ready probe gated on the background preload #408 edits the same file's healthcheck at
@@ -57,10 +57,11 @@, outsideeither block. docs(security): document opt-in revision pinning and digest verification #528's docs row and fix(serve): forward and validate per-request token budget overrides (… #566's both land in the
laya-servetable at :227, far from myConfiguration rows at :68, so
docs/docker.mdis clean.LAYA_MAX_CONCURRENT/LAYA_SHA256_DIGESTSserver names are also absent from the GPUcompose files, but they belong to
serve.py/revisions.pyand to the file docs(security): document opt-in revision pinning and digest verification #528 and feat(serve): add a /ready probe gated on the background preload #408 areediting, so they stay out of this PR.