Skip to content

fix(ts): normalise the action head at the model boundary, not in the decoder - #1195

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-ts-act-probability-boundary
Oct 14, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-ts-act-probability-boundary

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Moves the action head's softmax in laya-ts from the decode layer to the model boundary, and adds
laya-ts/tests/decode-parity.test.ts, which asserts 36 of the 40 answers in the shared decode
oracle laya-java/fixtures/decode.json field-for-field.

  • Agent._decodeRow now reports r4(actRow[0]) instead of r4(softmax(actRow)[0]).
  • The call site in _systemOneMany normalises the provider's row once: softmax(act[row] ?? [1, 0]).
  • 43 new tests: one per oracle case that the port can express, the permutation-invariant fields for
    the four option_order cases, and three on the probability-row contract itself.

Why

laya/agent.py:1329 (_forward) returns torch.softmax(act, -1) and Agent._decode_answers then
reads round(float(act[r, 0]), 4) (laya/agent.py:1365). laya/onnx_agent.py:805 normalises in
numpy for the same reason, and the Java decoder's parameter is literally actionProbabilities
(DecoderParityTest.java:71-72, asserted at tolerance 0.0). So the decode layer's input contract
is a probability row, in every port but this one: laya-ts softmaxed the row inside _decodeRow.

The consequence is measured, not argued. Feeding decode.json's own rows through _decodeRow —
the same call the Java parity test makes — 0 of its 40 cases agreed with the reference before
this change
, and every one of the 40 failed on the same field:

english/choice-2  action.act_probability  ts=0.6792  py=0.25     (identical on all 40 rows)

[0.25, -0.5] is the row the reference is handed (gen_fixtures.py:1854), and it reports 0.25;
the port reported softmax([0.25, -0.5])[0] = 0.6792. After the change 36 of 40 agree exactly,
including the padding, temperature-bucket, tie, uniform-20, ±700-magnitude and language-override
behaviours. The remaining 4 differ only on choice/probabilities/score, i.e. only on the
fields option_order moves — see What is deliberately not claimed below.

For the shipped ONNX providers nothing moves at all, and that is the point: providers.ts:574
selects the head's second output under the names ["act_logits", "act"], so those rows reach the
agent as the model emitted them, and one normalisation stays one normalisation ([3, 0] → 0.9526
before and after). The behaviour that changes is a provider that honours the decode contract
instead — one that hands over a probability row. [0.9, 0.1] was reported as 0.6899; it is now
reported as 0.9, as laya, the JVM port and the committed oracle all say.

How it was verified

cd laya-ts
NO_COLOR=1 npx --yes vitest run tests/decode-parity.test.ts   # 43 passed
NO_COLOR=1 npx --yes vitest run                               # branch: 929 tests, 928 passed

Base vs branch on the same checkout, same command (vitest run):

tree tests passed failed
upstream/main c73908d9 886 885 1
this branch a0d27f37 929 928 1

The one failure is identical in both rows and predates the branch:
load-robustness.test.ts > web provider downloads > passes signal to the model fetches and reports per-file progress, which needs installed node_modules to see a real AbortSignal identity —
npx --yes vitest here runs without them. It is not touched by this diff.

The 40-case differential is a scratch probe, run twice — once with laya-ts/src/agent.ts at
HEAD~1 (0 green, 40 red, all on action.act_probability) and once with the fix (36 green, 4 red,
none on action.act_probability). Dumps: /tmp/parity/decode_out_pre.json and
/tmp/parity/decode_out_ts.json. The probe compared numeric leaves at 1e-6 and is not committed.

The new suite itself is the red-at-base witness. With only laya-ts/src/agent.ts swapped back to
upstream/main's copy and the test file kept:

Test Files  1 failed (1)
     Tests  42 failed | 1 passed (43)

and the single test that passes at base is
predict() normalises the provider's action logits once, at the model boundary — the end-to-end
path through a provider that returns act_logits. So the two claims in Why are separate
measurements rather than one: the decode layer's contract is fixed (42 tests red → green), and the
shipped providers' numbers do not move (that test green → green).

tests/tsconfig.types.json includes only tests/types/**/*.ts, so it cannot reach this file;
node .../typescript/lib/tsc.js -p tsconfig.json --noEmit reports only the environmental
TS2688 Cannot find type definition file for 'node' (no @types/node in this checkout), and a
direct createProgram over src/*.ts plus tests/decode-parity.test.ts with ambient node types
produces no diagnostic in src/agent.ts.

Mutation check of the new suite — six arms on laya-ts/src/agent.ts, each anchored on a string
that occurs exactly once, the whole suite run per arm, and the one pre-existing failure subtracted
so a "kill" is a test the arm newly broke:

arm mutation newly failing tests
M1 softmax inside _decodeRow again (the pre-fix placement) 43 — every test in decode-parity.test.ts
M2 no normalisation at the model boundary 1 — predict() normalises the provider's action logits once, at the model boundary
M3 read actRow[1] instead of actRow[0] 43 — the whole file again
M4 three decimals instead of four 2 — the rounding test and the predict() test
M5 forget logitsRow.slice(0, k) 40 — every oracle case, i.e. the padding trap the family exists to catch
M6 ignore the per-language override 10 — 4 decode-parity cases plus 6 existing lang-temperature tests

kills=6 survivors=0; git status --porcelain after the harness: (clean). M5 is the arm worth
reading: with the slice gone, the 99.0 padding columns swallow the softmax and 40 of the 40
recorded answers break, which is what makes the padded rows in this fixture useful to assert
against.

ruff check laya/ … and python -m compileall -q laya/ tests/ were not run: this diff contains no
.py file. python laya-java/scripts/gen_fixtures.py was not run: no fixture is regenerated, and
decode.json is read as committed.

What is deliberately not claimed

option_order is not implemented in this port, and this PR does not pretend to fix it. The prompt
builder (agent.ts:605) passes no option order and _decodeRow performs no un-permutation, so a
question carrying option_order decodes in slot order while the reference un-permutes it
(laya/agent.py:1350, unpermute_probs at laya/common.py:557). The oracle's four permuted
cases are therefore asserted only on the three fields permutation cannot move (type,
confidence, answer_confidence) plus action, with the reason stated in the test header —
porting option_order end to end is a feature, not this fix, and it would have to touch the prefix
layer too.

docs/questions-and-answers.md:163-167 describes what act_probability means and says nothing
about where it is normalised, so no doc changes. The public API is unchanged: same field, same
name, same rounding.

Overlap with open PRs

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

…decoder

laya/agent.py::_forward returns torch.softmax(act, -1) and _decode_answers reads
act[r, 0] as a probability; laya/onnx_agent.py does the same in numpy and the Java
decoder's parameter is actionProbabilities. laya-ts softmaxed inside _decodeRow, so
the decode layer could not satisfy laya-java/fixtures/decode.json at all: 0 of its
40 cases agreed on action.act_probability. It now reads the row the way the
reference does, and tests/decode-parity.test.ts asserts 36 of the 40 answers
field-for-field (the four option_order cases stay open: the port has no
permutation step in either layer).
@NandhaKishorM
NandhaKishorM merged commit f80c7f6 into NandhaKishorM:main Oct 14, 2026
42 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.7, and #1196 carries the same fix.

Verified against the reference before merging: laya/agent.py::_forward returns torch.softmax(act.float(), -1), and laya/onnx_agent.py computes the same softmax in numpy, so by the time _decode_answers reads act[r, 0] the row is already a probability row. The port was softmaxing the raw row again inside _decodeRow, which is why [0.9, 0.1] reported 0.6899 where the reference reports 0.9.

Normalising once at the model boundary is the right place, because that is where the two Python backends do it and because the decode layer's contract (the Java decoder, and the act rows in laya-java/fixtures/decode.json) is a probability row. Reading actRow[0] directly afterwards also retires the slice(0, Math.max(2, ...)), which could only have padded a row validation had already refused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants