Skip to content

feat(laya-ts): report answer_confidence (max(p)) on every answer type - #386

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
nanami-0713:laya-ts-answer-confidence
Sep 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
nanami-0713:laya-ts-answer-confidence

Conversation

@nanami-0713

Copy link
Copy Markdown
Contributor

Closes the laya-ts half of the gap flagged in #361: the Python side has reported answer_confidence on every answer since #126, but laya-ts/src/ had nothing a threshold can read — the only confidence it reports is the entropy-based confidenceFromProbs, whose scale moves with the option count k and is not comparable across question types.

What this adds

  • answerConfidence(p) in laya-ts/src/common.ts — max(p) clipped to [0, 1], 1.0 for an empty distribution, matching laya/common.py:answer_confidence exactly.
  • Every decoded answer (choice / score / noul) in laya-ts/src/agent.ts now carries answer_confidence next to the existing confidence, same as laya/agent.py:_decode_answers. The existing confidence field is unchanged for current callers (entropy for choice/score, max(p_true, 1−p_true) for noul).
  • Exported from the package entry (laya-ts/src/index.ts), next to confidenceFromProbs.
  • For noul, answer_confidence is identical to confidence by construction (max over two options), with a comment saying so; it is still emitted so cross-type gating needs no special case.

Why max(p) and not the entropy value

The same argument as the #361 comment, now encoded as a test: with top probability 0.9 held fixed, confidenceFromProbs reports ~0.53 at k=2 and ~0.83 at k=77, so one fixed threshold abstains at a different effective probability per label-set size, while answerConfidence reads 0.9 in every row. This is also the quantity temperature scaling fits and the one the repository's calibration figures are computed on.

Tests

New laya-ts/tests/answer-confidence.test.ts (4 cases): direct max(p) + empty-distribution parity with Python; the k-invariance table from #361 (k = 2/5/20/77 at top-p 0.9) with the entropy divergence asserted alongside; and an end-to-end decode check through Agent.systemOne with a fake provider verifying answer_confidence equals the top of the emitted probabilities on all three answer types, and equals confidence on noul.

Full suite: 169/169 pass (13 files, vitest), tsc --noEmit clean. No behavior change for existing fields; no new dependencies.

The follow-on from #361 — a min_confidence-style abstention API in core + laya-ts — is deliberately not included; this PR only ships the quantity such a threshold should read, so the Python and TS sides agree before the gating API lands.

Parity with the Python side (#126): every decoded answer now carries
answer_confidence alongside the existing confidence, so callers can gate
across question types on a single calibrated number instead of the
entropy-based confidence, whose scale moves with the option count k.
Identical to confidence for noul (max(p) over two options); documented
divergence encoded in tests.

@aashish254 aashish254 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally at 3eb5039 (I opened #361, so this closes the TS half of a gap I care about — checked it closely):

  • Full suite 169/169 (13 files, vitest), tsc --noEmit clean.
  • Exact parity with laya/common.py:answer_confidence: same empty-distribution → 1.0, same [0, 1] clip. At the call site p = softmax(logits[r].slice(0, k)), so max(...p) here is precisely Python's max(p[:k]) — no padding can leak into the max.
  • The noul branch emitting answer_confidence identical to confidence (max over two options) is right, and keeping it unconditional means the future cross-type gate needs no special case.
  • The k-invariance table test (top-p 0.9 at k = 2/5/20/77, entropy divergence asserted alongside) encodes exactly the argument from #361 / #327 as a regression test — nice.

No behavior change to existing fields, no new dependencies. This is the quantity the min_confidence follow-on from #361 should read, and landing it first keeps Python and TS in agreement before any gating API ships — I'm happy to build the TS threshold on top of this once it merges. LGTM.

@nanami-0713

Copy link
Copy Markdown
Contributor Author

Thank you for the close read and for running the suite locally — and especially for spelling out the call-site equivalence: since p comes from softmax(logits[r].slice(0, k)), max(...p) is max(p[:k]) with no padding path into the max. I had asserted the parity from the Python side but not from the TS slicing side, so that confirmation closes the loop.

Please do build the min_confidence threshold on top — that was exactly why this PR kept the field unconditional on noul and changed nothing about the existing confidence, so the gating API can read one number across all three types without special cases. Happy to review or co-test the TS side when you get to it.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, this closes a real parity gap: TypeScript users now get the same calibrated answer_confidence on every answer type that Python reports. Leaving the existing confidence field alone keeps it fully backward compatible, and the test that holds the top probability fixed while the option count changes explains the design well. Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants