Repository navigation
Close Jev parity gaps: fix criteria crash, add typed SDK surface, patterns library, and GPU-validated scaling guidance - #2
Conversation
render_options assumed string criteria throughout. Dict values in choice and
score criteria stringified as Python reprs (single-quoted, not JSON), and dict
values in noul criteria raised TypeError on string concatenation.
Add render_criterion, which passes strings through untouched and json.dumps
anything else, and use it in all three branches. This enables rubric-style
options such as {"what": ..., "not_for": [...], "examples": [...]}.
Also plumb truncation reporting through build_sequence: an on_truncate callback
fires with the kept and total token counts when state does not fit, so callers
can surface dropped input rather than losing it silently.
Adds the first tests to the repo, covering the rendering branches and the
left/right truncation paths.
Callers previously wrote code against raw dicts and caught bare ValueError, RuntimeError and FileNotFoundError, which made it impossible to distinguish a malformed question from a missing checkpoint. laya.errors adds ten classes under LayaError. Each keeps its original builtin as a secondary base, so existing `except ValueError:` handlers keep working. laya.types adds ChoiceAnswer, ScoreAnswer, NoulAnswer, Response and supporting types. They expose attributes (answer.choice, answer.confidence) while implementing Mapping, so answer["choice"], .get(), `in` and ** unpacking all still behave as before. Response also carries request_id and a list of Truncation records. Includes a module hygiene test that walks the AST of every laya module for names used but never bound, which catches missing imports on error paths that normal tests do not reach.
Wire the typed errors and answers through the inference runtime: - Validate question definitions up front. Unknown types, missing keys and criteria of the wrong shape now raise a specific QuestionError subclass naming the offending question id, instead of surfacing as a KeyError or a confusing failure further down. - Return Response with typed answers, a request_id, and a truncated list recording which questions lost state and how many tokens went missing. - Expose max_len, head_max_len and truncate_left as constructor arguments. Warn when max_len exceeds the length the checkpoint was trained at, since sequence building will succeed there while accuracy and calibration are unverified. - Raise DeviceError and InferenceError on the device and forward-pass failure paths, which makes them retryable, and route every warning through the laya logger instead of print().
AsyncAgent wraps Agent and runs the forward pass in a thread executor. Laya inference is local compute rather than network I/O, so the point is keeping the event loop responsive and letting callers fan out with asyncio.gather, not overlapping waits. batch() evaluates many states under a concurrency limit. RetryPolicy provides exponential backoff with jitter. It retries only InferenceError by default: question and configuration errors are deterministic, so retrying them would just fail identically while hiding the real cause.
Library-level compositions over the existing predict path, with no model changes. Each helper takes anything exposing system_one(state, questions), so they work with Agent, AsyncAgent or a test stub. confidence_gate splits answers into automatic and escalated by calibrated confidence. route dispatches on a choice answer. composite_score combines questions into one weighted 0-1 score. self_consistency asks a question several ways and flags disagreement. rerank sorts candidates. cascade runs staged questions with early exit. fan_out evaluates many states. two_stage_choice and taxonomy_choice select among more options than fit a single head. chunk_state and map_reduce cover documents longer than the context window by chunking and combining per-chunk answers.
Evaluating two_stage_choice against labelled data on a GPU showed it performed substantially worse than a single call wherever both fit: 0.229 against 0.750 at 100 options. Decomposing the error put stage 1 at 0.042 and stage 2 at 1.000, so the model was picking within groups almost perfectly and picking between groups at close to random. Two causes, both fixed here: - Group descriptions listed every member, which at 400 options consumed the whole head budget and left nothing for the state. Logs showed "kept 0 of 19 tokens", meaning the document was discarded entirely and accuracy was 0.000. Descriptions are now length-bounded with an "and N more" suffix. - More groups than fit a single head were passed as one question. Group selection now recurses when the group count exceeds group_size. The deeper finding is that default chunking produces groups named group_0, group_1 and so on, which carry no signal for the first stage to choose between. Passing group_of so groups match real categories scores 0.646 against 0.229. The docstring now says to supply it, with the measured numbers. Adds the eval and training scripts used, so the results can be reproduced.
Completes the pattern set against the parity analysis, which named function
calling and citation checking.
select_tool picks which tool should handle a request. It adds a decline option
so the model is not forced to choose, gates on confidence so a caller can
refuse to fire a tool on a coin flip, accepts OpenAI-style schema dicts, and
routes through two_stage_choice once the tool list outgrows a single head.
check_citation asks whether a source supports a claim, contradicts it, or is
off-topic. Two findings shaped the implementation:
- Passing {"claim": ..., "source": ...} as a JSON state and referencing the
fields from the instructions scored 0.048 on a supported pair, against 0.865
for the same content as inline premise/hypothesis text. The checkpoint does
not resolve backtick field references for this kind of question, so the
inline framing is used.
- The contradiction signal is unreliable: an unrelated source scored 0.996,
higher than a genuine refutation at 0.965. A relevance question separates
most such cases and downgrades borderline contradictions, but cannot separate
all of them. This is documented rather than tuned away, since any threshold
fitting the remaining case on a handful of examples would be overfitting.
The guidance is that a "supported" verdict is dependable and anything else
needs review.
Export the error classes, typed answers, AsyncAgent, RetryPolicy and the patterns module from the package root. Add a test extra for pytest. The release is backward compatible: typed answers implement Mapping, and every new error keeps its original builtin base, so existing callers and the fine-tuning notebooks continue to work unchanged.
The GPU report records what was measured on an L40S rather than what was predicted, including the results that contradicted the original analysis. Context window: fine-tuning at 8192 tokens moved accuracy from 0.594 to 1.000 and ECE from 0.350 to 0.000 in 50 steps, at 2.9s per step and 9.7GB peak memory. Both training and evaluation data were synthetic and drawn from the same generator, so this establishes feasibility and cost, not production readiness; the report says so and the numbers should not be published as benchmark results. A production checkpoint needs real long-document data. Option ceiling: measured at roughly 126 options with a real tokenizer, not the 27 to 38 originally estimated, because build_sequence shrinks per-option budgets when they do not fit. Guidance is now to use a single call below about 100 options and two_stage_choice with explicit grouping above that. Context length sweep: included, with a note that it does not show what it was designed to show. Accuracy sits near chance at every length including 512, so it measures task difficulty rather than degradation with context. 32k remains unrecommended. ModernBERT was pre-trained to 1024 tokens and extended to 8192; forward passes run beyond that, but needle representations drift from 0.95 to 0.81 cosine similarity by 26k and no RoPE scaling is configured. The README gains sections on structured criteria, typed answers, error handling, async, patterns and the context window, plus a note that the comparison table against Jev reports each system's published characteristics and is not a head-to-head measurement.
The repo has five open PRs from external contributors and no CI. PR #2 found a real crash in shipped code that no test would have caught: render_options raises TypeError on a noul question whose criteria values are dicts, and leaks Python reprs into the prompt for choice/score. Verified against main. ci.yml tests on py3.9/3.11/3.13 (CPU torch, under a minute), ruff, and a package build with twine check security.yml gitleaks on full history, pip-audit, CodeQL (security-extended), plus two repo-specific gates: the shipped package must never pickle/torch.load (checkpoints come from the Hub, so that path is remote code execution) and must never shell out or exec release.yml push a vX.Y.Z tag -> tests, version-matches-tag check, PyPI via trusted publishing (no token in secrets), GitHub Release dependabot weekly pip and github-actions updates Also: drop two unused imports flagged by the new lint gate, and cover guess_latin_language directly rather than importing it unused (71 -> 77 tests). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
render_options assumed every criterion value was a string. A noul question
whose criteria were dicts raised
TypeError: can only concatenate str (not "dict") to str
and choice/score stringified dicts as Python reprs ({'desc': 'payments'}),
putting a Python literal into the prompt instead of JSON. Dict criteria are a
documented feature, so this was a hard failure on ordinary input.
Reported and originally fixed by @trocker in #2; taking the render_criterion
approach from that PR on its own, since #2 also carries ~3.7k lines of
unrelated work.
Also fixes a quieter bug in the same expression: the choice branch tested
'if not v', so a criterion whose value was 0 or False was rendered as though
it had no description. Only None and "" mean missing now.
tests/test_criteria.py: 29 regression tests covering the reported crash, JSON
rather than repr for dict/list/number criteria, 0 and False as real values,
non-ASCII, unserialisable objects, and that string criteria are unchanged.
Both suites now run in CI and in the release gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@NandhaKishorM - What are your thoughts on this? |
|
Thanks for the extensive work here. Since main has diverged and criteria crash handling already landed, could you rebase against latest main and consider splitting this into smaller focused PRs? |
|
Closing this, eight days after asking for the split, and with thanks rather than as a dismissal. This was opened very early and it found real things. Each of the P0 items has since landed, in focused PRs that were reviewable on their own:
So: if there is a part of this that today's Closing on staleness, not on merit. |
What changed
P0 — Blocking
Structured JSON criteria (
d5d7e9f).render_optionsassumed strings throughout. Dictcriteria stringified as Python reprs for
choice/score, and raisedTypeError: can only concatenate str (not "dict") to strfornoul— a hard crash on adocumented Jev feature. New
render_criterionpasses strings through andjson.dumpseverything else, in all three branches.
Silent truncation (
8b309ed).truncate_leftexisted but was never plumbed throughpredict, so long inputs lost their tail with no warning. It's now a constructor argumentalongside
max_len, truncation is logged, andResponse.truncatedreports exactly whichquestions lost how many tokens.
Option ceiling (
44b3632).two_stage_choice/taxonomy_choiceselect among moreoptions than fit a single head. See the measured caveat below — this one did not go as
planned.
P1 — SDK ergonomics
537b4f8)LayaError, each keeping its builtin base soexcept ValueError:still works537b4f8)ChoiceAnswer.confidence,NoulAnswer.is_true,Response.request_id— all implementingMapping, soanswer["choice"]and**answerstill behave34cafce)AsyncAgent,batch(max_concurrency=…),RetryPolicythat retries only transientInferenceError8b309ed)print()replaced with thelayalogger;request_idon each responseP2 — Patterns (
3089310,1ac1676)14 helpers in
laya.patterns, compositions overpredictwith no model changes:confidence_gate,route,composite_score,self_consistency,rerank,cascade,fan_out,two_stage_choice,taxonomy_choice,chunk_state,map_reduce,select_tool,check_citation,check_citations.