Skip to content

Close Jev parity gaps: fix criteria crash, add typed SDK surface, patterns library, and GPU-validated scaling guidance - #2

Closed
trocker wants to merge 9 commits into
NandhaKishorM:mainfrom
trocker:feat/jev-parity-v0.2.0
Closed

trocker wants to merge 9 commits into
NandhaKishorM:mainfrom
trocker:feat/jev-parity-v0.2.0

Conversation

@trocker

@trocker trocker commented Sep 18, 2026

Copy link
Copy Markdown

What changed

P0 — Blocking

Structured JSON criteria (d5d7e9f). render_options assumed strings throughout. Dict
criteria stringified as Python reprs for choice/score, and raised
TypeError: can only concatenate str (not "dict") to str for noul — a hard crash on a
documented Jev feature. New render_criterion passes strings through and json.dumps
everything else, in all three branches.

Silent truncation (8b309ed). truncate_left existed but was never plumbed through
predict, so long inputs lost their tail with no warning. It's now a constructor argument
alongside max_len, truncation is logged, and Response.truncated reports exactly which
questions lost how many tokens.

Option ceiling (44b3632). two_stage_choice / taxonomy_choice select among more
options than fit a single head. See the measured caveat below — this one did not go as
planned.

P1 — SDK ergonomics

Typed errors (537b4f8) 10 classes under LayaError, each keeping its builtin base so except ValueError: still works
Typed answers (537b4f8) ChoiceAnswer.confidence, NoulAnswer.is_true, Response.request_id — all implementing Mapping, so answer["choice"] and **answer still behave
Async + retries (34cafce) AsyncAgent, batch(max_concurrency=…), RetryPolicy that retries only transient InferenceError
Logging (8b309ed) Every print() replaced with the laya logger; request_id on each response

P2 — Patterns (3089310, 1ac1676)

14 helpers in laya.patterns, compositions over predict with no model changes:
confidence_gate, route, composite_score, self_consistency, rerank, cascade,
fan_out, two_stage_choice, taxonomy_choice, chunk_state, map_reduce,
select_tool, check_citation, check_citations.


teja added 9 commits September 18, 2026 16:09
render_options assumed string criteria throughout. Dict values in choice and
score criteria stringified as Python reprs (single-quoted, not JSON), and dict
values in noul criteria raised TypeError on string concatenation.

Add render_criterion, which passes strings through untouched and json.dumps
anything else, and use it in all three branches. This enables rubric-style
options such as {"what": ..., "not_for": [...], "examples": [...]}.

Also plumb truncation reporting through build_sequence: an on_truncate callback
fires with the kept and total token counts when state does not fit, so callers
can surface dropped input rather than losing it silently.

Adds the first tests to the repo, covering the rendering branches and the
left/right truncation paths.
Callers previously wrote code against raw dicts and caught bare ValueError,
RuntimeError and FileNotFoundError, which made it impossible to distinguish a
malformed question from a missing checkpoint.

laya.errors adds ten classes under LayaError. Each keeps its original builtin
as a secondary base, so existing `except ValueError:` handlers keep working.

laya.types adds ChoiceAnswer, ScoreAnswer, NoulAnswer, Response and supporting
types. They expose attributes (answer.choice, answer.confidence) while
implementing Mapping, so answer["choice"], .get(), `in` and ** unpacking all
still behave as before. Response also carries request_id and a list of
Truncation records.

Includes a module hygiene test that walks the AST of every laya module for
names used but never bound, which catches missing imports on error paths that
normal tests do not reach.
Wire the typed errors and answers through the inference runtime:

- Validate question definitions up front. Unknown types, missing keys and
  criteria of the wrong shape now raise a specific QuestionError subclass
  naming the offending question id, instead of surfacing as a KeyError or
  a confusing failure further down.
- Return Response with typed answers, a request_id, and a truncated list
  recording which questions lost state and how many tokens went missing.
- Expose max_len, head_max_len and truncate_left as constructor arguments.
  Warn when max_len exceeds the length the checkpoint was trained at, since
  sequence building will succeed there while accuracy and calibration are
  unverified.
- Raise DeviceError and InferenceError on the device and forward-pass failure
  paths, which makes them retryable, and route every warning through the laya
  logger instead of print().
AsyncAgent wraps Agent and runs the forward pass in a thread executor. Laya
inference is local compute rather than network I/O, so the point is keeping
the event loop responsive and letting callers fan out with asyncio.gather,
not overlapping waits. batch() evaluates many states under a concurrency limit.

RetryPolicy provides exponential backoff with jitter. It retries only
InferenceError by default: question and configuration errors are deterministic,
so retrying them would just fail identically while hiding the real cause.
Library-level compositions over the existing predict path, with no model
changes. Each helper takes anything exposing system_one(state, questions), so
they work with Agent, AsyncAgent or a test stub.

confidence_gate splits answers into automatic and escalated by calibrated
confidence. route dispatches on a choice answer. composite_score combines
questions into one weighted 0-1 score. self_consistency asks a question several
ways and flags disagreement. rerank sorts candidates. cascade runs staged
questions with early exit. fan_out evaluates many states.

two_stage_choice and taxonomy_choice select among more options than fit a
single head. chunk_state and map_reduce cover documents longer than the
context window by chunking and combining per-chunk answers.
Evaluating two_stage_choice against labelled data on a GPU showed it performed
substantially worse than a single call wherever both fit: 0.229 against 0.750
at 100 options. Decomposing the error put stage 1 at 0.042 and stage 2 at
1.000, so the model was picking within groups almost perfectly and picking
between groups at close to random.

Two causes, both fixed here:

- Group descriptions listed every member, which at 400 options consumed the
  whole head budget and left nothing for the state. Logs showed "kept 0 of 19
  tokens", meaning the document was discarded entirely and accuracy was 0.000.
  Descriptions are now length-bounded with an "and N more" suffix.
- More groups than fit a single head were passed as one question. Group
  selection now recurses when the group count exceeds group_size.

The deeper finding is that default chunking produces groups named group_0,
group_1 and so on, which carry no signal for the first stage to choose between.
Passing group_of so groups match real categories scores 0.646 against 0.229.
The docstring now says to supply it, with the measured numbers.

Adds the eval and training scripts used, so the results can be reproduced.
Completes the pattern set against the parity analysis, which named function
calling and citation checking.

select_tool picks which tool should handle a request. It adds a decline option
so the model is not forced to choose, gates on confidence so a caller can
refuse to fire a tool on a coin flip, accepts OpenAI-style schema dicts, and
routes through two_stage_choice once the tool list outgrows a single head.

check_citation asks whether a source supports a claim, contradicts it, or is
off-topic. Two findings shaped the implementation:

- Passing {"claim": ..., "source": ...} as a JSON state and referencing the
  fields from the instructions scored 0.048 on a supported pair, against 0.865
  for the same content as inline premise/hypothesis text. The checkpoint does
  not resolve backtick field references for this kind of question, so the
  inline framing is used.
- The contradiction signal is unreliable: an unrelated source scored 0.996,
  higher than a genuine refutation at 0.965. A relevance question separates
  most such cases and downgrades borderline contradictions, but cannot separate
  all of them. This is documented rather than tuned away, since any threshold
  fitting the remaining case on a handful of examples would be overfitting.
  The guidance is that a "supported" verdict is dependable and anything else
  needs review.
Export the error classes, typed answers, AsyncAgent, RetryPolicy and the
patterns module from the package root. Add a test extra for pytest.

The release is backward compatible: typed answers implement Mapping, and every
new error keeps its original builtin base, so existing callers and the
fine-tuning notebooks continue to work unchanged.
The GPU report records what was measured on an L40S rather than what was
predicted, including the results that contradicted the original analysis.

Context window: fine-tuning at 8192 tokens moved accuracy from 0.594 to 1.000
and ECE from 0.350 to 0.000 in 50 steps, at 2.9s per step and 9.7GB peak
memory. Both training and evaluation data were synthetic and drawn from the
same generator, so this establishes feasibility and cost, not production
readiness; the report says so and the numbers should not be published as
benchmark results. A production checkpoint needs real long-document data.

Option ceiling: measured at roughly 126 options with a real tokenizer, not the
27 to 38 originally estimated, because build_sequence shrinks per-option
budgets when they do not fit. Guidance is now to use a single call below about
100 options and two_stage_choice with explicit grouping above that.

Context length sweep: included, with a note that it does not show what it was
designed to show. Accuracy sits near chance at every length including 512, so
it measures task difficulty rather than degradation with context.

32k remains unrecommended. ModernBERT was pre-trained to 1024 tokens and
extended to 8192; forward passes run beyond that, but needle representations
drift from 0.95 to 0.81 cosine similarity by 26k and no RoPE scaling is
configured.

The README gains sections on structured criteria, typed answers, error
handling, async, patterns and the context window, plus a note that the
comparison table against Jev reports each system's published characteristics
and is not a head-to-head measurement.
NandhaKishorM added a commit that referenced this pull request Sep 19, 2026
The repo has five open PRs from external contributors and no CI. PR #2 found a
real crash in shipped code that no test would have caught:
render_options raises TypeError on a noul question whose criteria values are
dicts, and leaks Python reprs into the prompt for choice/score. Verified
against main.

ci.yml       tests on py3.9/3.11/3.13 (CPU torch, under a minute), ruff, and a
             package build with twine check
security.yml gitleaks on full history, pip-audit, CodeQL (security-extended),
             plus two repo-specific gates: the shipped package must never
             pickle/torch.load (checkpoints come from the Hub, so that path is
             remote code execution) and must never shell out or exec
release.yml  push a vX.Y.Z tag -> tests, version-matches-tag check, PyPI via
             trusted publishing (no token in secrets), GitHub Release
dependabot   weekly pip and github-actions updates

Also: drop two unused imports flagged by the new lint gate, and cover
guess_latin_language directly rather than importing it unused (71 -> 77 tests).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Sep 19, 2026
render_options assumed every criterion value was a string. A noul question
whose criteria were dicts raised
  TypeError: can only concatenate str (not "dict") to str
and choice/score stringified dicts as Python reprs ({'desc': 'payments'}),
putting a Python literal into the prompt instead of JSON. Dict criteria are a
documented feature, so this was a hard failure on ordinary input.

Reported and originally fixed by @trocker in #2; taking the render_criterion
approach from that PR on its own, since #2 also carries ~3.7k lines of
unrelated work.

Also fixes a quieter bug in the same expression: the choice branch tested
'if not v', so a criterion whose value was 0 or False was rendered as though
it had no description. Only None and "" mean missing now.

tests/test_criteria.py: 29 regression tests covering the reported crash, JSON
rather than repr for dict/list/number criteria, 0 and False as real values,
non-ASCII, unserialisable objects, and that string criteria are unchanged.
Both suites now run in CI and in the release gate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM NandhaKishorM reopened this Sep 19, 2026
@trocker

trocker commented Sep 19, 2026

Copy link
Copy Markdown
Author

@NandhaKishorM - What are your thoughts on this?

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for the extensive work here. Since main has diverged and criteria crash handling already landed, could you rebase against latest main and consider splitting this into smaller focused PRs?

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you again for the early, extensive work. Several pieces of it have since landed in focused PRs: the criteria crash, typed question validation (#183), and a Jev-compatible server (#31). If any part is still missing on today's main, a small focused PR for it would be very welcome.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing this, eight days after asking for the split, and with thanks rather than as a dismissal.

This was opened very early and it found real things. Each of the P0 items has since landed, in focused PRs that were reviewable on their own:

main has moved by 64 more pull requests since the last comment here, so the branch now conflicts across laya/agent.py, laya/common.py, laya/__init__.py and the docs, and the conflicts are mostly in code that was rewritten by the work this PR anticipated. A rebase would be reconstructing the diff rather than replaying it, which is more work than re-deriving whichever piece is still missing.

So: if there is a part of this that today's main still does not do, a small PR for that one thing is very welcome and I will review it quickly. The candidates I would look at from your original list are the truncate_left plumbing and Response.truncated, since truncation reporting has changed several times since (#668 and #689 both touched it) and I am not certain your version of it is covered.

Closing on staleness, not on merit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants