Repository navigation
feat(laya-ts): report answer_confidence (max(p)) on every answer type - #386
Conversation
Parity with the Python side (#126): every decoded answer now carries answer_confidence alongside the existing confidence, so callers can gate across question types on a single calibrated number instead of the entropy-based confidence, whose scale moves with the option count k. Identical to confidence for noul (max(p) over two options); documented divergence encoded in tests.
aashish254
left a comment
There was a problem hiding this comment.
Verified locally at 3eb5039 (I opened #361, so this closes the TS half of a gap I care about — checked it closely):
- Full suite 169/169 (13 files, vitest),
tsc --noEmitclean. - Exact parity with
laya/common.py:answer_confidence: same empty-distribution → 1.0, same [0, 1] clip. At the call sitep = softmax(logits[r].slice(0, k)), somax(...p)here is precisely Python'smax(p[:k])— no padding can leak into the max. - The noul branch emitting
answer_confidenceidentical toconfidence(max over two options) is right, and keeping it unconditional means the future cross-type gate needs no special case. - The k-invariance table test (top-p 0.9 at k = 2/5/20/77, entropy divergence asserted alongside) encodes exactly the argument from #361 / #327 as a regression test — nice.
No behavior change to existing fields, no new dependencies. This is the quantity the min_confidence follow-on from #361 should read, and landing it first keeps Python and TS in agreement before any gating API ships — I'm happy to build the TS threshold on top of this once it merges. LGTM.
|
Thank you for the close read and for running the suite locally — and especially for spelling out the call-site equivalence: since Please do build the |
|
Thank you, this closes a real parity gap: TypeScript users now get the same calibrated answer_confidence on every answer type that Python reports. Leaving the existing confidence field alone keeps it fully backward compatible, and the test that holds the top probability fixed while the option count changes explains the design well. Merging. |
Closes the laya-ts half of the gap flagged in #361: the Python side has reported
answer_confidenceon every answer since #126, butlaya-ts/src/had nothing a threshold can read — the only confidence it reports is the entropy-basedconfidenceFromProbs, whose scale moves with the option count k and is not comparable across question types.What this adds
answerConfidence(p)inlaya-ts/src/common.ts—max(p)clipped to [0, 1],1.0for an empty distribution, matchinglaya/common.py:answer_confidenceexactly.choice/score/noul) inlaya-ts/src/agent.tsnow carriesanswer_confidencenext to the existingconfidence, same aslaya/agent.py:_decode_answers. The existingconfidencefield is unchanged for current callers (entropy for choice/score, max(p_true, 1−p_true) for noul).laya-ts/src/index.ts), next toconfidenceFromProbs.noul,answer_confidenceis identical toconfidenceby construction (max over two options), with a comment saying so; it is still emitted so cross-type gating needs no special case.Why max(p) and not the entropy value
The same argument as the #361 comment, now encoded as a test: with top probability 0.9 held fixed,
confidenceFromProbsreports ~0.53 at k=2 and ~0.83 at k=77, so one fixed threshold abstains at a different effective probability per label-set size, whileanswerConfidencereads 0.9 in every row. This is also the quantity temperature scaling fits and the one the repository's calibration figures are computed on.Tests
New
laya-ts/tests/answer-confidence.test.ts(4 cases): directmax(p)+ empty-distribution parity with Python; the k-invariance table from #361 (k = 2/5/20/77 at top-p 0.9) with the entropy divergence asserted alongside; and an end-to-end decode check throughAgent.systemOnewith a fake provider verifyinganswer_confidenceequals the top of the emittedprobabilitieson all three answer types, and equalsconfidenceon noul.Full suite: 169/169 pass (13 files, vitest),
tsc --noEmitclean. No behavior change for existing fields; no new dependencies.The follow-on from #361 — a
min_confidence-style abstention API in core + laya-ts — is deliberately not included; this PR only ships the quantity such a threshold should read, so the Python and TS sides agree before the gating API lands.