Repository navigation
fix(ts): normalise the action head at the model boundary, not in the decoder - #1195
Conversation
…decoder laya/agent.py::_forward returns torch.softmax(act, -1) and _decode_answers reads act[r, 0] as a probability; laya/onnx_agent.py does the same in numpy and the Java decoder's parameter is actionProbabilities. laya-ts softmaxed inside _decodeRow, so the decode layer could not satisfy laya-java/fixtures/decode.json at all: 0 of its 40 cases agreed on action.act_probability. It now reads the row the way the reference does, and tests/decode-parity.test.ts asserts 36 of the 40 answers field-for-field (the four option_order cases stay open: the port has no permutation step in either layer).
|
Merged in 0.4.7, and #1196 carries the same fix. Verified against the reference before merging: Normalising once at the model boundary is the right place, because that is where the two Python backends do it and because the decode layer's contract (the Java decoder, and the |
What
Moves the action head's softmax in
laya-tsfrom the decode layer to the model boundary, and addslaya-ts/tests/decode-parity.test.ts, which asserts 36 of the 40 answers in the shared decodeoracle
laya-java/fixtures/decode.jsonfield-for-field.Agent._decodeRownow reportsr4(actRow[0])instead ofr4(softmax(actRow)[0])._systemOneManynormalises the provider's row once:softmax(act[row] ?? [1, 0]).the four
option_ordercases, and three on the probability-row contract itself.Why
laya/agent.py:1329(_forward) returnstorch.softmax(act, -1)andAgent._decode_answersthenreads
round(float(act[r, 0]), 4)(laya/agent.py:1365).laya/onnx_agent.py:805normalises innumpy for the same reason, and the Java decoder's parameter is literally
actionProbabilities(
DecoderParityTest.java:71-72, asserted at tolerance0.0). So the decode layer's input contractis a probability row, in every port but this one:
laya-tssoftmaxed the row inside_decodeRow.The consequence is measured, not argued. Feeding
decode.json's own rows through_decodeRow—the same call the Java parity test makes — 0 of its 40 cases agreed with the reference before
this change, and every one of the 40 failed on the same field:
[0.25, -0.5]is the row the reference is handed (gen_fixtures.py:1854), and it reports0.25;the port reported
softmax([0.25, -0.5])[0] = 0.6792. After the change 36 of 40 agree exactly,including the padding, temperature-bucket, tie, uniform-20, ±700-magnitude and language-override
behaviours. The remaining 4 differ only on
choice/probabilities/score, i.e. only on thefields
option_ordermoves — see What is deliberately not claimed below.For the shipped ONNX providers nothing moves at all, and that is the point:
providers.ts:574selects the head's second output under the names
["act_logits", "act"], so those rows reach theagent as the model emitted them, and one normalisation stays one normalisation (
[3, 0]→0.9526before and after). The behaviour that changes is a provider that honours the decode contract
instead — one that hands over a probability row.
[0.9, 0.1]was reported as0.6899; it is nowreported as
0.9, aslaya, the JVM port and the committed oracle all say.How it was verified
Base vs branch on the same checkout, same command (
vitest run):upstream/mainc73908d9a0d27f37The one failure is identical in both rows and predates the branch:
load-robustness.test.ts > web provider downloads > passes signal to the model fetches and reports per-file progress, which needs installednode_modulesto see a realAbortSignalidentity —npx --yes vitesthere runs without them. It is not touched by this diff.The 40-case differential is a scratch probe, run twice — once with
laya-ts/src/agent.tsatHEAD~1(0 green, 40 red, all onaction.act_probability) and once with the fix (36 green, 4 red,none on
action.act_probability). Dumps:/tmp/parity/decode_out_pre.jsonand/tmp/parity/decode_out_ts.json. The probe compared numeric leaves at 1e-6 and is not committed.The new suite itself is the red-at-base witness. With only
laya-ts/src/agent.tsswapped back toupstream/main's copy and the test file kept:and the single test that passes at base is
predict() normalises the provider's action logits once, at the model boundary— the end-to-endpath through a provider that returns
act_logits. So the two claims in Why are separatemeasurements rather than one: the decode layer's contract is fixed (42 tests red → green), and the
shipped providers' numbers do not move (that test green → green).
tests/tsconfig.types.jsonincludes onlytests/types/**/*.ts, so it cannot reach this file;node .../typescript/lib/tsc.js -p tsconfig.json --noEmitreports only the environmentalTS2688 Cannot find type definition file for 'node'(no@types/nodein this checkout), and adirect
createProgramoversrc/*.tsplustests/decode-parity.test.tswith ambient node typesproduces no diagnostic in
src/agent.ts.Mutation check of the new suite — six arms on
laya-ts/src/agent.ts, each anchored on a stringthat occurs exactly once, the whole suite run per arm, and the one pre-existing failure subtracted
so a "kill" is a test the arm newly broke:
_decodeRowagain (the pre-fix placement)decode-parity.test.tspredict() normalises the provider's action logits once, at the model boundaryactRow[1]instead ofactRow[0]predict()testlogitsRow.slice(0, k)lang-temperaturetestskills=6 survivors=0;git status --porcelainafter the harness:(clean). M5 is the arm worthreading: with the slice gone, the
99.0padding columns swallow the softmax and 40 of the 40recorded answers break, which is what makes the padded rows in this fixture useful to assert
against.
ruff check laya/ …andpython -m compileall -q laya/ tests/were not run: this diff contains no.pyfile.python laya-java/scripts/gen_fixtures.pywas not run: no fixture is regenerated, anddecode.jsonis read as committed.What is deliberately not claimed
option_orderis not implemented in this port, and this PR does not pretend to fix it. The promptbuilder (
agent.ts:605) passes no option order and_decodeRowperforms no un-permutation, so aquestion carrying
option_orderdecodes in slot order while the reference un-permutes it(
laya/agent.py:1350,unpermute_probsatlaya/common.py:557). The oracle's four permutedcases are therefore asserted only on the three fields permutation cannot move (
type,confidence,answer_confidence) plusaction, with the reason stated in the test header —porting
option_orderend to end is a feature, not this fix, and it would have to touch the prefixlayer too.
docs/questions-and-answers.md:163-167describes whatact_probabilitymeans and says nothingabout where it is normalised, so no doc changes. The public API is unchanged: same field, same
name, same rounding.
Overlap with open PRs
agent.tshunks@@ -21,7and@@ -837,6) and feat(laya-ts): batch inference, tensor-plumbing perf fixes, and DX helpers #340 (agent.tshunks spanning@@ -300,6…@@ -471,6) both edit this file, and neither reaches the lines this change edits(
@@ -686,7,@@ -720,7,@@ -739,8). No textual conflict either way.@@ -447,7) is likewise outside these ranges.decideBatchgrows a decode row, it should call the same boundary-normalised helper;nothing here prevents that.
Checklist
main