Skip to content

fix(sdk): align TypeScript confidence contract with HTTP responses - #1072

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
debjyotikm:fix/sdk-jev-confidence
Oct 11, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
debjyotikm:fix/sdk-jev-confidence

Conversation

@debjyotikm

Copy link
Copy Markdown
Contributor

What

Align the TypeScript SDK with the existing HTTP x_jev_confidence extension:

  • Add optional numeric fields to choice and score answer types, and validate finite values in [0, 1] when present. Older responses that omit the extension remain accepted; noul has no Jev confidence.
  • Preserve exact integration-test equality for all core prediction fields. Independently calculate the expected extension from direct Python probabilities and check the server's four-decimal rounding before excluding only that field from core parity.
  • Add positive, negative, type-level, and parity-regression tests. Pin the offline fixture to the default, non-strict HTTP contract.

No production Python inference behavior, confidence formulas, model weights, or dependencies change.

Why

The server now enriches HTTP answers with x_jev_confidence, while direct Agent.predict() results do not include this HTTP-layer field. The SDK integration test compared the complete objects and therefore failed despite identical underlying predictions.

This reproduces on upstream main at 6880462 and is independent of the residual-adapter feature:

The fix does not disable response equality or use the server's enrichment helper as its own expected-value oracle. Regression cases ensure that changed probabilities, decisions, usage, unexpected fields, and incorrect or missing extensions still fail.

Verification

Local CPU validation on Node 22.23.2 and 24.21.0, each:

cd sdk/typescript
npm ci
npm test
npm run test:integration
npm pack --dry-run

Before: the live HTTP integration test failed on the extra confidence fields. After: 29 SDK tests and 2 integration tests passed, with no skips, on each Node version. TypeScript build/type checking and package dry-run passed on both versions. The integration fixture uses a tiny offline checkpoint, not downloaded weights.

Repository-root checks also passed:

python tests/test_hooks_api.py
python -m pytest tests/test_serve.py -q
python tests/test_confidence.py
python sdk/typescript/scripts/sync_presets.py --check
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
zensical build --strict --clean
git diff --check
  • API contract: 579 checks passed.
  • HTTP server suite: 264 passed.
  • Confidence suite: 214 passed.
  • Presets, lint, compilation, and strict docs build passed; no griffe: diagnostics.

These are local results; GitHub CI on the submitted commit is separate.

Scope

One maintenance fix based on upstream 6880462 (0.4.2). No architecture features, private reports, datasets, checkpoint artifacts, or unrelated formatting changes are included. The three feature PRs can rebase after this fix lands.

@aashish254

Copy link
Copy Markdown
Contributor

Short answer: yours is the only open pull request whose TypeScript SDK lanes are green on a base that already contains c932543, so merging it first is what turns the repo's red signal back into a useful one. Here is the confirmation, with numbers rather than a request.

What I measured across the 65 currently open PRs:

One point worth keeping straight, because the job names are close: TypeScript (node20) / (node22) run npm test -- --run in laya-ts (.github/workflows/ci.yml:26) and are green everywhere, including on main. The red pair is the one in sdk/typescript, which is your directory.

I also read main's own failing job (run 38041040045, job 114181166375) to look for anything that contradicts your diagnosis. There isn't any, and it corroborates it: the unit suite there reports tests 27 / pass 27 / fail 0, and the failing run is a single test (tests 1 / pass 0 / fail 1) -- sdk/typescript/test/integration.mjs:46, JavaScript → HTTP → Python Router → real offline Agent inference. Its printed diff is one key: the SDK side carries x_jev_confidence (refund 1, team 0.007, urgency 0) and the fixture's direct-Python expectation does not, with every other field equal. That is the field c932543 adds on the HTTP layer, exactly as your "Why" section describes. Computing the expected extension from the direct-Python probabilities and then checking the server's four-decimal rounding, instead of exempting the whole object, is the variant of this fix that keeps the parity assertion worth having.

The practical consequence for maintainers is why I am posting instead of staying quiet. All ten of my open PRs -- #1092 #1093 #1095 #1096 #1098 #1086 #1085 #1084 #1083 #1074 -- carry head commits dated 2026-10-10, i.e. after c932543, and each of them fails exactly two checks with zero non-SDK failures (35 or 36 passing, 1 or 2 skipping depending on the job). Today each of those reads as "2 checks failed" for a reason that is not in its own diff, and a reviewer has to open the job log to discover that. Merging #1072 first means the next push to any branch -- or a plain re-run of its checks -- shows zero failures, without anyone touching a line of code.

@aashish254 aashish254 mentioned this pull request Oct 11, 2026
4 tasks done
@NandhaKishorM
NandhaKishorM merged commit 0a44379 into NandhaKishorM:main Oct 11, 2026
38 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 11, 2026
Reverts 5e7a88b ("test(sdk): include x_jev_confidence in test server
expected fixture"), carried on #1113's branch. #1072 and that commit are two
mutually exclusive fixes for the same red lane, and they were each green in
isolation. Together they are not: #1072's assertPredictionParity strips
x_jev_confidence from the HTTP answer before comparing, so an `expected` that
also carries it leaves the field only on the expected side and
deepStrictEqual fails on all three choice/score answers.

Keeping #1072's side on purpose. It derives the extension from the direct
model's probabilities and checks the server's four-decimal rounding against
that, so the test fails if _add_jev_confidence is wrong. Building `expected`
by calling _add_jev_confidence puts the same helper on both sides of the
assertion, where any error in it cancels out.

#1113's own subject, the bare "questions" question id in laya/cli.py, is
untouched by this revert.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 11, 2026
35 pull requests from 11 contributors. Headline #1072: sdk/typescript's
response validation rejected the x_jev_confidence that 0.4.2's #1066 started
sending, so the TypeScript SDK lane has been red on main since that release.

Also #1073 (LAYA_TOURNAMENT_GROUP), #1083 (laya.jev_confidence), #1105
(model: null across all four MCP tools), #1104, #1109, #1112, #1113,
#1108 and #1117 (laya-ts), #1090 (Java floors remeasured on the merged
tree), and 20 documentation and test-gate corrections.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.3. This was the one that mattered most this pass, because the lane it fixes had been red on main for two days and was hiding everything else.

What I measured on the merged tree, sdk/typescript:

  • npm test -> 29 passed, 0 failed.
  • npm run test:integration -> 2 passed, 0 failed, including JavaScript -> HTTP -> Python Router -> real offline Agent inference. On 68804629 that test fails with deepStrictEqual on three answers, x_jev_confidence present on the HTTP side and absent on the direct-inference side.
  • npm pack --dry-run -> clean, 36 files.

Your assertPredictionParity is the reason I kept this fix rather than the other one proposed for the same lane. Deriving the extension from the direct model's probabilities and then checking the server's four-decimal rounding against that number means the test fails if _add_jev_confidence is wrong. The alternative, building expected by calling _add_jev_confidence on the direct result, puts the same helper on both sides of the assertion, so any error in it cancels out and the test still passes.

One thing you should know, because it affected your PR without anything being wrong in it. #1113 carried a second fix for this same lane on its branch, commit 5e7a88b2, which changed tests/sdk_server_fixture.py to expected = _add_jev_confidence(agent.predict(state, questions)). Each PR was green alone. Merged together they are not: your helper strips the field from the HTTP answer before comparing, so an expected that also carries it leaves the field on the expected side only, and all three choice/score answers fail. I reverted 5e7a88b2 on the integration branch and kept your side, as a separate commit with that reasoning in its message. #1113's own subject, the bare questions question id in the CLI, is untouched.

Rest of the gate on the merged tree: 87 of 87 Python suites pass, laya-ts 870 tests across 35 files, ruff clean on laya/, and a 94-row real-weights decision diff over both checkpoints, predict, predict_long and Router, identical to 68804629.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants