Skip to content

docs(examples): example 35 must teach the MPS autocast policy it denies - #842

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:example-mps-amp-policy
Oct 2, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:example-mps-amp-policy

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/35_device_selection_and_latency.py told the reader that mixed precision is a CUDA thing
and that MPS and CPU run fp32, in two places: the banner, and a line printed under its own timings.
Both are contradicted by laya/agent.py. The example now prints the policy it actually runs under,
and tests/test_runtime_fixes.py holds the page's sentences to the branch that sets them.

The banner said:

Precision follows the device -- mixed precision (bf16/fp16) is a CUDA win, while CPU and MPS run
fp32; torch.autocast has no MPS backend here, so Laya only wraps the forward pass on CUDA.

and after timing both legs it printed:

both legs used fp32 (device.type in ('cpu','mps') -> torch.float32);
mixed precision would only be enabled on CUDA.

The second one is half true, which is how it survived: that call asks 3 questions, and 3 is below
mps_amp_min_rows, so MPS really did run fp32. The rule it states is not why. Agent.__init__'s mps
branch sets amp_enabled = True at torch.float16, and _amp_enabled_for(rows) decides per call:

elif self.device.type == "mps":
    self.amp_enabled = True
    self.dtype = torch.float16

So the page was teaching a device rule where the real rule is a batch-size gate -- on the one device
(Apple Silicon) whose owners most need to know that fp16 is available at all.

The example now reports the policy per leg (amp_enabled, dtype, dtype_for(rows),
mps_amp_min_rows), then measures three batch sizes against three thresholds, and observes the
precision from inside a real forward -- torch.is_autocast_enabled() and the first Linear layer's
output dtype, recorded by a forward hook -- rather than relabeling dtype, which is what let the old
sentence be printed with a straight face while dtype said torch.float16 and the call ran fp32.

Why

A reader who takes the page at face value concludes their Mac cannot do mixed precision, and never
discovers LAYA_MPS_AMP_MIN_ROWS -- a knob that exists precisely because MPS fp16 is a batch-size
decision. Nothing in CI catches the sentence: examples are not executed by the test job, and no test
read the page's prose, so the claim could only rot.

The gate reads the contract out of the source instead of restating it, the way the example gates in
tests/test_truncation.py do:

  • The device branches of Agent.__init__ are located through ast and compared with ast.unparse,
    so a reformat of the branch moves the gate, not the test. Two arms prove this: single-quoting
    == 'mps' and collapsing the branch body to one line both leave the suite green.
  • Sentences are policed, not whole strings: a sentence may still say an MPS call runs fp32, but only
    if it names mps_amp_min_rows or dtype_for -- the distinction the page had backwards. A sentence
    may confine mixed precision to CUDA only if it names another device the code enables it on.
  • float16 is taken as MPS's target from the branch, in a window that stops at the next device's
    name, so the page cannot satisfy "MPS ... bfloat16" by going on to describe XPU.
  • The threshold stated in prose is pinned to MPS_AMP_MIN_ROWS_DEFAULT; the one printed is read off
    the agent (ast Load, twice -- reported and obeyed), not hand-copied.
  • Every LAYA_* the page names must be one the module really reads. LAYA_DEVICE is allowed because
    the page's own sentence says Laya does not read it.
  • The two sentences quoted above are kept verbatim in the test as controls, so a rule that matches
    nothing cannot pass for being empty.

How it was verified

python -B tests/test_runtime_fixes.py
python -B examples/35_device_selection_and_latency.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

tests/test_runtime_fixes.py went from 67 checks on main to 86 here (+19, 86 passed, 0 failed).
ruff reports "All checks passed!" and compileall is clean; no laya/ file is touched.

The example on a 32 GB Apple Silicon Mac, torch 2.14.0, multilingual checkpoint from the local Hub
cache, MPS available, CUDA not:

   torch 2.14.0   cuda_available=False   mps_available=True
   explicit selection example: device = os.environ.get('LAYA_DEVICE') or None

   == CPU ==
   cpu  -> loaded on cpu  dtype=torch.float32 median    62.5 ms/call  20.8 ms/question
        `amp_enabled`=False   `dtype_for(3 rows)`=torch.float32   `mps_amp_min_rows`=5 (MPS only)

   == MPS ==
   mps  -> loaded on mps  dtype=torch.float16 median    31.6 ms/call  10.5 ms/question
        `amp_enabled`=True    `dtype_for(3 rows)`=torch.float32   `mps_amp_min_rows`=5 (MPS only)

   speed-up: CPU 62.5 ms -> MPS 31.6 ms  (1.97x)
   answers agree after JSON serialisation: True

   1 question   as shipped      `mps_amp_min_rows`=5          autocast=False torch.float32     14.2 ms
   1 question   fp16 forced on  `mps_amp_min_rows`=1          autocast=True  torch.float16     17.3 ms
   1 question   fp32 forced off `mps_amp_min_rows`=1000000000 autocast=False torch.float32     13.2 ms
                this machine, fp16 over fp32 at this batch size: 1.31x
   3 questions  as shipped      `mps_amp_min_rows`=5          autocast=False torch.float32     26.7 ms
   3 questions  fp16 forced on  `mps_amp_min_rows`=1          autocast=True  torch.float16     23.7 ms
   3 questions  fp32 forced off `mps_amp_min_rows`=1000000000 autocast=False torch.float32     26.1 ms
                this machine, fp16 over fp32 at this batch size: 0.91x
   5 questions  as shipped      `mps_amp_min_rows`=5          autocast=True  torch.float16     32.4 ms
   5 questions  fp16 forced on  `mps_amp_min_rows`=1          autocast=True  torch.float16     30.0 ms
   5 questions  fp32 forced off `mps_amp_min_rows`=1000000000 autocast=False torch.float32     35.7 ms
                this machine, fp16 over fp32 at this batch size: 0.84x

That is the whole argument in the reader's own numbers: MPS really does run torch.float16 once the
batch reaches the gate (is_autocast_enabled() is set inside the forward and the first Linear
emits fp16), and the shipped threshold sits where fp16 stops paying for itself. The medians are
repeats on one machine, so the page prints the ratio instead of asserting a crossover: a second
independent run of the same arms on the same laptop gave 1.33x / 0.90x / 0.88x, and the CPU/MPS
speed-up there was 1.92x.

The gate was mutation-checked over 16 arms, each restoring both files from a byte snapshot and
requiring the suite's completion line (a crash or an aborted run is scored ABORT, not a kill):

OK       base                                                            failed=0 []
OK       M1 mps branch stops enabling autocast                           killed by 1/1: ['amp-policy/mixed precision is not confined to CUDA']
OK       M2 mps target moves to bfloat16                                 killed by 2/2: ['amp-policy/mps autocasts at float16', "35/states MPS's target as the branch sets it (bfloat16)"]
OK       M3 mps branch keeps no dtype target                             killed by 1/1: ['amp-policy/mps autocasts at float16']
OK       M4 the per-call row gate is dropped                             killed by 1/1: ['mps-gate/one row stays fp32', 'mps-gate/below threshold stays fp32', 'dtype_for/mps below threshold is fp32']
OK       M5 the shipped MPS threshold changes                            killed by 1/1: ['35/the MPS threshold it states is the module default']
OK       M6 the runtime stops reading LAYA_MPS_AMP_MIN_ROWS              killed by 1/1: ['mps-gate/env override', 'amp-policy/the runtime reads an env var per AMP knob', '35/names no env var the runtime does not read, or says so']
OK       M7 xpu target moves to float16                                  killed by 1/1: ['amp-policy/xpu autocasts at bfloat16']
OK       M8 the banner lie comes back                                    killed by 1/1: ['35/carries no sentence that misstates the autocast policy']
OK       M9 the printed lie comes back under the timings                 killed by 1/1: ['35/carries no sentence that misstates the autocast policy']
OK       M10 the MPS fp32 sentence stops naming the key                  killed by 1/1: ['35/carries no sentence that misstates the autocast policy']
OK       M11 the threshold loses the backticks a reader greps for        killed by 1/1: ['35/names `mps_amp_min_rows` so a reader can grep for it']
OK       M12 the forward is no longer observed                           killed by 1/1: ['35/observes the precision inside a real forward']
OK       M13 the threshold is hand-copied instead of read off the agent  killed by 1/1: ["35/reads the agent's threshold rather than hand-copying it"]
OK       control: the mps branch test is single-quoted                   failed=0 []
OK       control: the mps branch body is collapsed to one line           failed=0 []

16 arms, 0 not as expected

M10 is the arm that matters most: it leaves the sentence true ("the MPS leg ran fp32 because 3 is
under the threshold") and fails only because it stops naming the key a reader would have to look up.
M4 and M6 are killed by the runtime checks that already existed plus the new prose checks, so the
page and the library fail together rather than the page alone.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #

No issue: found by reading examples/*.py claims against laya/agent.py. No public API changed --
the example only now prints what the runtime already does, and the new checks are in the suite CI
already runs (python tests/test_runtime_fixes.py).

… sets

The page told the reader mixed precision is a CUDA win "while CPU and MPS run
**fp32**", that "torch.autocast has no MPS backend here", and printed under its
own timings that "both legs used fp32 (device.type in ('cpu','mps') ->
torch.float32)". The mps branch of Agent.__init__ sets amp_enabled = True at
torch.float16 and _amp_enabled_for enables it per call at mps_amp_min_rows rows,
so the rule was wrong even though the observation -- a 3-question call running
fp32 -- happened to be right.

Example 35 now prints the policy it measured: amp_enabled, dtype and
dtype_for(rows) per leg, then a table of three batch sizes against three
thresholds, observing torch.is_autocast_enabled() and the first Linear layer's
output dtype from inside a real forward. On an M-series Mac at the multilingual
checkpoint this run: fp16 1.33x the fp32 time at one question, 0.88x at five,
with the shipped gate taking fp32 below 5 rows and fp16 at 5.
…ets it

Reads the autocast policy out of Agent.__init__ through ast.unparse, so a
reformat of the branch moves the gate and not the test, and polices the
example's sentences: one may say an MPS call runs fp32, but only if it names
mps_amp_min_rows or dtype_for, which is the distinction the page used to get
backwards. The stated threshold is pinned to MPS_AMP_MIN_ROWS_DEFAULT, MPS's
autocast target to the branch that sets it (in a window that stops at the next
device's name, so a page cannot describe XPU and pass for MPS), every LAYA_*
named to a read the module really does, and the two sentences the gate exists
for are kept verbatim as controls so a rule that matches nothing cannot pass for
being empty.

15 checks, 86 in the suite, all green, mutation-checked over 16 arms: 13 kills
(both lies, the branch, the target, the gate, the threshold, the env read, the
backticks, the hand-copied number, the dropped forward hook) and 2 controls that
stay green across a reformat of the branch.
@NandhaKishorM
NandhaKishorM merged commit 2f55c54 into NandhaKishorM:main Oct 2, 2026
23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants