Repository navigation
docs(examples): example 35 must teach the MPS autocast policy it denies - #842
Merged
NandhaKishorM merged 2 commits intoOct 2, 2026
Merged
Conversation
… sets
The page told the reader mixed precision is a CUDA win "while CPU and MPS run
**fp32**", that "torch.autocast has no MPS backend here", and printed under its
own timings that "both legs used fp32 (device.type in ('cpu','mps') ->
torch.float32)". The mps branch of Agent.__init__ sets amp_enabled = True at
torch.float16 and _amp_enabled_for enables it per call at mps_amp_min_rows rows,
so the rule was wrong even though the observation -- a 3-question call running
fp32 -- happened to be right.
Example 35 now prints the policy it measured: amp_enabled, dtype and
dtype_for(rows) per leg, then a table of three batch sizes against three
thresholds, observing torch.is_autocast_enabled() and the first Linear layer's
output dtype from inside a real forward. On an M-series Mac at the multilingual
checkpoint this run: fp16 1.33x the fp32 time at one question, 0.88x at five,
with the shipped gate taking fp32 below 5 rows and fp16 at 5.
…ets it Reads the autocast policy out of Agent.__init__ through ast.unparse, so a reformat of the branch moves the gate and not the test, and polices the example's sentences: one may say an MPS call runs fp32, but only if it names mps_amp_min_rows or dtype_for, which is the distinction the page used to get backwards. The stated threshold is pinned to MPS_AMP_MIN_ROWS_DEFAULT, MPS's autocast target to the branch that sets it (in a window that stops at the next device's name, so a page cannot describe XPU and pass for MPS), every LAYA_* named to a read the module really does, and the two sentences the gate exists for are kept verbatim as controls so a rule that matches nothing cannot pass for being empty. 15 checks, 86 in the suite, all green, mutation-checked over 16 arms: 13 kills (both lies, the branch, the target, the gate, the threshold, the env read, the backticks, the hand-copied number, the dropped forward hook) and 2 controls that stay green across a reformat of the branch.
4 tasks done
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/35_device_selection_and_latency.pytold the reader that mixed precision is a CUDA thingand that MPS and CPU run fp32, in two places: the banner, and a line printed under its own timings.
Both are contradicted by
laya/agent.py. The example now prints the policy it actually runs under,and
tests/test_runtime_fixes.pyholds the page's sentences to the branch that sets them.The banner said:
and after timing both legs it printed:
The second one is half true, which is how it survived: that call asks 3 questions, and 3 is below
mps_amp_min_rows, so MPS really did run fp32. The rule it states is not why.Agent.__init__'s mpsbranch sets
amp_enabled = Trueattorch.float16, and_amp_enabled_for(rows)decides per call:So the page was teaching a device rule where the real rule is a batch-size gate -- on the one device
(Apple Silicon) whose owners most need to know that fp16 is available at all.
The example now reports the policy per leg (
amp_enabled,dtype,dtype_for(rows),mps_amp_min_rows), then measures three batch sizes against three thresholds, and observes theprecision from inside a real forward --
torch.is_autocast_enabled()and the firstLinearlayer'soutput dtype, recorded by a forward hook -- rather than relabeling
dtype, which is what let the oldsentence be printed with a straight face while
dtypesaidtorch.float16and the call ran fp32.Why
A reader who takes the page at face value concludes their Mac cannot do mixed precision, and never
discovers
LAYA_MPS_AMP_MIN_ROWS-- a knob that exists precisely because MPS fp16 is a batch-sizedecision. Nothing in CI catches the sentence: examples are not executed by the test job, and no test
read the page's prose, so the claim could only rot.
The gate reads the contract out of the source instead of restating it, the way the example gates in
tests/test_truncation.pydo:Agent.__init__are located throughastand compared withast.unparse,so a reformat of the branch moves the gate, not the test. Two arms prove this: single-quoting
== 'mps'and collapsing the branch body to one line both leave the suite green.if it names
mps_amp_min_rowsordtype_for-- the distinction the page had backwards. A sentencemay confine mixed precision to CUDA only if it names another device the code enables it on.
float16is taken as MPS's target from the branch, in a window that stops at the next device'sname, so the page cannot satisfy "MPS ... bfloat16" by going on to describe XPU.
MPS_AMP_MIN_ROWS_DEFAULT; the one printed is read offthe agent (
astLoad, twice -- reported and obeyed), not hand-copied.LAYA_*the page names must be one the module really reads.LAYA_DEVICEis allowed becausethe page's own sentence says Laya does not read it.
nothing cannot pass for being empty.
How it was verified
tests/test_runtime_fixes.pywent from 67 checks onmainto 86 here (+19,86 passed, 0 failed).ruffreports "All checks passed!" andcompileallis clean; nolaya/file is touched.The example on a 32 GB Apple Silicon Mac, torch 2.14.0, multilingual checkpoint from the local Hub
cache, MPS available, CUDA not:
That is the whole argument in the reader's own numbers: MPS really does run
torch.float16once thebatch reaches the gate (
is_autocast_enabled()is set inside the forward and the firstLinearemits fp16), and the shipped threshold sits where fp16 stops paying for itself. The medians are
repeats on one machine, so the page prints the ratio instead of asserting a crossover: a second
independent run of the same arms on the same laptop gave 1.33x / 0.90x / 0.88x, and the CPU/MPS
speed-up there was 1.92x.
The gate was mutation-checked over 16 arms, each restoring both files from a byte snapshot and
requiring the suite's completion line (a crash or an aborted run is scored ABORT, not a kill):
M10 is the arm that matters most: it leaves the sentence true ("the MPS leg ran fp32 because 3 is
under the threshold") and fails only because it stops naming the key a reader would have to look up.
M4 and M6 are killed by the runtime checks that already existed plus the new prose checks, so the
page and the library fail together rather than the page alone.
Checklist
mainFixes #
No issue: found by reading
examples/*.pyclaims againstlaya/agent.py. No public API changed --the example only now prints what the runtime already does, and the new checks are in the suite CI
already runs (
python tests/test_runtime_fixes.py).