Repository navigation
feat(java): dispatch router lifecycle hooks, and add AsyncHook - #1046
Conversation
Routing decides whether a state goes to the English checkpoint or the
multilingual one, and the English checkpoint collapses to near-random on a
script it has no tokens for (0.100 on Hindi at 20 options, where random is
0.050). So this is the part of the port where a small difference is a wrong
answer rather than a rounding discrepancy.
What the reference leans on, and what Java gives instead:
* four regexes. Java's \w is not Python's -- it admits combining marks and
join controls -- and \d and \p{L} follow the JDK's Unicode version rather
than CPython's. Each pattern is a hand-written scan over the recorded
tables. The identifier scan reproduces the reference's lookbehind, which
is what keeps it linear: without it a 4,000-character token costs 205 ms
against 0.45 ms for prose, and a state is user input.
* str.lower(). Not String.toLowerCase, which applies the contextual
final-sigma rule; and not Character.toLowerCase either, which at Unicode
13.0 does not know the lowercase of 40 code points CPython 15.0 maps
(all of Vithkuqi among them) and returns them unchanged. A table, so the
mapping does not depend on the JDK a consumer runs.
* str.isspace(), which strip() and split() work on. Character.isWhitespace
rejects four of its members -- U+0085, U+00A0, U+2007, U+202F -- and
Character.isSpaceChar is wrong the other way. A no-break space pasted
from a word processor otherwise fuses two words into one token that no
stopword list holds.
* dict insertion order, which is data here: it orders the public script
profile and breaks every tie between two languages with the same score.
LinkedHashMap throughout, and the fixture now records the stopword order
explicitly -- it was being sorted away, which is the same mistake the
generator's missing sort_keys was fixed for once already.
Faster than the reference and than laya-ts, on the same corpus, best-of-7
after three warmups:
Python laya-ts laya-java
1,700 analyse calls 85 ms 34 ms 25 ms
20 calls on a 3.8 KB state 37 ms 14 ms 8 ms
Two changes earn most of that, and neither costs a divergence. Script lookup
is a binary search over one sorted table instead of a scan over 25 scripts in
order -- equivalent because no two ranges overlap, which the generator now
refuses to emit a table without, and which a test proves by sweeping every
code point. And each character predicate answers ASCII from a pair of 64-bit
masks computed from the same predicate as its range table, instead of ten-odd
comparisons. Counting letters by script now tallies into an int[] with no
boxing or hashing per character.
Gated by 137 tests. The corpus is 106 cases recorded from the reference, each
asserting all eight output fields and, for a string, all fourteen
intermediate steps, so a failure names the stage that broke. The character
tables are compared against CPython by a sha256 over all 1,114,112 code
points, nine digests in 500 bytes of fixture -- replacing a hand check that
proved the tables once and would not have noticed a JDK upgrade moving one.
22 mutants were run against the suite. 19 die. The three survivors are
equivalent today for a reason that is now a guarded invariant rather than a
coincidence: no script range reaches below U+0370, so the two Latin cuts
cannot differ; none of the 40 code points whose case the JDK disputes can
start a non-Latin run; and no empty line can be selected, so keeping trailing
empty fields on the newline split changes nothing. Each has a test that fails
if the enabling property does. The rounding mode needed the same treatment
the other way: it is unobservable through analyse at an even threshold, so a
half-up mutant passed the whole corpus, and the rule is now pinned by a
fixture and the unobservability itself asserted.
Also fixes a cliff rather than waiting for it: with every table in its field
initialiser the class stopped compiling with "code too large" once the
lowercase map was added, because array literals share one <clinit> and a JVM
method body may not exceed 65,535 bytes. Each table now has its own factory
method, and the generator refuses to emit one too large for that budget
instead of letting javac report it as a size error.
378 tests pass, including the end-to-end parity against the real graph.
`laya.presets` ships five question sets for the workflows laya is used for --
support triage, email triage, LLM guardrails, moderation, and model routing.
They are data, not code, and one changed word is a different question put to
the model and therefore a different answer. So they are compared against the
reference on what the model is actually shown -- the instruction, the rendered
options in order, the labels, the legend -- rather than on the Java object.
`stateField` comes with them, and it is the part a port gets wrong quietly. A
preset says which part of the state it reads by naming it in backticks ("What
does the customer want in `message`?") and this reads it back out, which is
what keeps the preset and the field name in step with no hand-maintained
table. Three details are load-bearing and now recorded:
* Python's \w, not Java's. Java's admits combining marks and join controls,
and `Character.isLetterOrDigit` excludes the underscore -- which alone
turns `customer_message` from one field into none.
* the scan resumes after the closing backtick, as findall does. In
"`a`b`c`" the fields are a and c; a scanner that resumed inside its own
match would offer b as well and turn a one-field preset into a
three-field one, which answers null.
* only the instruction is read. Criteria quote user-facing labels, so a
backtick in one is punctuation; scanning the whole rendered question
would report a field the question does not read.
The fixture grew the cases that pin all of that -- a spec naming nothing, one
naming two fields, one naming the same field twice, non-word backticks, and a
backtick confined to a criterion -- plus the preset that takes an argument.
That one holds a trap: `email_questions({})` returns the DEFAULT categories,
because the reference tests the argument for truthiness and an empty dict is
falsy. It reads like a way to ask for no categories and is not, and a port
that checked only for null would build a choice question with no options and
throw.
Two of the recorded hostile cases give an instruction of null or omit it.
They are not expressible here: `Question` refuses a blank instruction at
construction, earlier than the reference does. The test asserts that refusal
rather than skipping the case, so the difference is checked instead of
silently uncovered.
23 tests. 12 mutants were run against them -- a reworded instruction, a
reworded criterion, two options swapped, a score level reordered, a dropped
question, the noul sides swapped, an unordered criteria map, a shared
(cached) preset map, and each of the three stateField details above. All 12
die.
254 tests pass.
…uilt from
A route explains itself in a `reason` string, and that string reaches API
responses. Two of them are built with Python formats that Java spells
differently, and one of them interpolates the mixed segment -- a slice of the
caller's own text -- so neither difference is theoretical.
`%r` is CPython's repr. The quote is a single quote unless the string holds one
and no double quote; a backslash and the quote are escaped; newline, carriage
return and tab get letter escapes; and anything `str.isprintable()` calls
unprintable becomes a numeric escape in lowercase hex. That last rule is the
one an ASCII-only check misses, and it is not a corner: U+00A0 NO-BREAK SPACE
is unprintable to CPython and arrives in pasted text constantly, so stopping
at U+007F writes a raw control character into a response. U+00AD, U+200B,
U+FEFF, U+202E and the private-use and unassigned planes are the same story,
while every printable non-ASCII character -- CJK, emoji, a combining mark --
is deliberately left alone.
That needed one more recorded table, `str.isprintable()`, because the boundary
is 148,998 printable code points against 965,114 unprintable ones and it moves
with the Unicode version. It is gated the way the others are: a sha256 over
`repr` of every code point in Unicode, so the escape rule is proven
exhaustively rather than on the cases someone thought to list. The listed
cases are still there for the rules a reader can name -- all four quote
combinations, the letter escapes, the three widths of numeric escape.
`%.0f` rounds halves to EVEN. `String.format("%.0f", 12.5)` gives "13" where
CPython gives "12", and a share of one letter in eight is exactly 12.5%, so
the router would misreport a percentage it puts in front of a user. Also
reachable at 0.205 -> "20" and 0.625 -> "62". `Math.rint` is the right
primitive and needs no BigDecimal.
Both also carry an ASCII fast path for the new table, generated from the same
predicate as the table itself.
258 tests pass. The percent test additionally counts how many of its own cases
would come out differently under `String.format`, and fails if that count is
small -- otherwise it would be asserting that two agreeing implementations
agree.
Three checkpoints, and choosing between them is not a tuning question. The
English checkpoint does not degrade gently off English, it collapses: 0.100 on
20-option MASSIVE Hindi intent against 0.050 for random guessing, and it
reports high confidence while doing so (ECE 0.855). The multilingual
checkpoint gains 21 points on non-English XNLI and gives up about 6 on English
suites. So a wrong route is not a slightly worse answer, it is a confident
wrong one -- which is why every precedence level and every detection branch
has a recorded case here.
This commit is the decision and the registry. It runs no model and reads no
disk, so it is safe on every request and testable without a checkpoint; a test
pins that by routing through a Router pointed at paths that do not exist. The
load-and-evict lifecycle follows separately.
Precedence, highest first: an explicit model, an explicit task, a detected
typed-decisions workflow (opt-in only -- that checkpoint is fine-tuned on four
synthetic workflows and must not be a silent default), an explicit language, a
per-call hint, the router's installed hint, detection, then the default.
The `reason` strings are asserted, not just the model. They reach API
responses, and they are built from the two Python formats this port added last
commit: `repr`, and a `%.0f` that rounds halves to EVEN. Both see arbitrary
input, because one reason interpolates the mixed segment -- a slice of the
caller's own text. Two route cases exist only to pin that: seventy Latin
letters against ten Greek ones is a non-Latin share of exactly 10/80, so the
reason reads "12% of letters" and not "13%", and five accented characters in
exactly forty is the same halfway case on the other percentage. Without them
the router could misreport a percentage shown to a user and every other case
would still pass. A third carries a no-break space and a single quote inside
the mixed segment, so the reason has to switch to double quotes AND escape a
non-printable character in one string.
Two API notes. A hint is one functional interface rather than the reference's
"code or callable": `LanguageHint.of("pt")` for a fixed code, a lambda for a
language-identification model. Returning null abstains, and abstaining is a
real answer -- it falls through to detection instead of pinning a checkpoint
on no evidence, which is also what `LANG=C` in a minimal container gives.
And `Question.spec()` is new, because a schema signature has to serialise a
question the way the reference's dictionary holds it; its javadoc states the
one thing a typed question cannot carry, which is a caller's own extra keys or
an unusual key order inside `criteria`.
129 tests on the router, 388 in the suite. 18 mutants were run: 17 die. The
survivor is a finding rather than a gap -- the reference rewrites a task
spelled `typed_decisions` before resolving it, and that rewrite cannot change
an answer, because name resolution lowercases first and the alias table
already holds both spellings. The faithful branch is kept and a test proves
the redundancy, so it fails if either spelling ever leaves that table.
Also fixed a trap in my own test: a blank dynamic-test display name aborts the
whole factory, and an aborted container is not a failed test -- 38 recorded
cases silently never ran while the suite stayed green.
`load`, `lease`, `attach`, `preload`, `unload`, `loaded`, `predict` and
`close`, with an LRU cap of two resident checkpoints as in the reference.
Loading goes through a factory seam rather than a hard-coded directory layout:
the reference downloads from the hub, this port takes local paths, so the
mapping from a checkpoint to an Agent belongs to the caller --
`checkpointsRoot(path)` for the obvious layout, `agents(factory)` for anything
else. A seam for the same reason `InferenceSession` is one: the parts most
likely to be wrong here are which agent eviction picks, whether an attached
one is ever closed, and whether two threads asking for the same cold
checkpoint pay for two copies, and none of that has anything to do with ONNX.
Through a stub they are tested in milliseconds and run in every CI job instead
of only the parity lane.
ONE PLACE THIS CANNOT COPY THE REFERENCE, and it is why `Lease` exists. There,
eviction drops a reference and CPython's refcounting keeps the agent alive for
whoever still holds it. A JVM will not free a native ONNX session that way, so
the router has to close it -- and closing one that a caller is mid-prediction
with would crash that prediction. So eviction marks a slot retired and closes
it when the last lease is released; `predict` leases internally, so the
ordinary path is safe without a caller thinking about it. An agent handed in
through `attach` is never closed at all: the caller kept ownership.
Two bugs found while writing this, both in code I had just written:
* `acquire` registered one in-flight handle and then passed a different,
unregistered one to `build`. Every concurrent waiter would have blocked on
a latch nobody ever counted down -- a deadlock, not a duplicate build, so
no amount of counting loads would have surfaced it.
* a test asserted `"multilingual"` equals `"multilingual"`, which is true
whatever the router does. It now writes a checkpoint whose `pt`
temperature override makes the same question answer differently with and
without the language, asserts the two differ, and only then asserts the
router passed the one it detected.
20 lifecycle tests. 16 mutants, all 16 die, and two of them were worth the
trouble for what they revealed rather than for dying:
* removing the cache fast path in `acquire` passed everything. The fast path
is not an optimisation: it returns a resident checkpoint without touching
the build lock, so a request for a loaded model is not stalled behind a
cold load of a different one -- in a server, the difference between one
checkpoint loading and all traffic stopping. There is a test for that now.
* removing the in-flight latch also passed, because the build lock plus the
re-check already guarantee one build on the success path. The latch earns
its place on the FAILURE path: without it each waiter re-enters the
factory to discover the same failure, which for a load that fails by
timing out is six slow failures instead of one. The javadoc said the latch
was what prevented duplicate builds; that was wrong and now says what it
actually does.
The second of those also caught a flaw in the test harness: the stub failed a
build before waiting on its gate, so a failing build finished before a second
caller arrived and nothing was ever shared. The test was measuring the stub.
A real load that fails takes as long as one that succeeds, and the stub now
does too.
408 tests pass. The CI floors move with them, so a suite that aborts instead
of running still fails the lane.
Every option of a choice shares one `head_max_len`, so a seventeen-label set
leaves a few tokens per label and the model is shown truncated options.
Shortlisting embeds the state and each option, keeps the top k, and runs ONE
prediction over the reduced set. No second decision pass: the model still
scores every criterion it is handed, there are just fewer of them.
`Predictor` comes with it -- one interface, `predict(state, questions)`,
implemented by both `Agent` and `Router` with no change to either, since both
already had that exact signature. The reference accepts anything with
`predict` or `system_one`; this is the typed form of the same idea, and it is
what lets a caller shortlist identically whether the questions go to a fixed
checkpoint or a routed one. The .NET port has `ILayaPredictor` for the same
reason.
AN API GAP THE TESTS FOUND. My first cut took only a `Question`, and a
`Question` always carries a non-blank instruction -- while the reference takes
criteria and an OPTIONAL instruction separately, and its default is to embed
the state alone. So thirteen recorded rankings diverged, and not subtly: the
test had to invent a placeholder instruction, which changed every query text.
There is now a criteria overload with a nullable instruction, which is the
reference's actual signature.
WHAT CANNOT BE MATCHED BIT FOR BIT, measured rather than assumed. The
reference scores with `np.dot(matrix, vector)`, a BLAS matrix-vector product,
and how BLAS blocks that accumulation differs between implementations: on
Apple Accelerate one row of a four-row product differs in the last bit from
the same row computed alone. Every intermediate -- the dot, both norms, the
division -- agrees bit for bit between numpy and a sequential loop; it is the
blocking of the matrix form that does not.
That has a consequence beyond parity: recorded verbatim, these scores would
make the fixture disagree with ITSELF between a developer's macOS and CI's
OpenBLAS, and the drift gate would fail for a reason unrelated to any port. So
the fixture records twelve decimals -- four orders of magnitude tighter than
any porting error this repository has seen, four orders looser than the 1e-16
BLAS spread -- and regenerating twice now produces identical bytes. The LABEL
ORDER stays exact, which is the only thing a consumer observes and the thing a
ranking bug actually moves. A test guards the tolerance from widening: it
fails if the recorded precision and the comparison stop matching.
47 tests. 19 mutants, all 19 die -- but two needed real work rather than a
tick, and both were hidden by that very tolerance:
* removing the zero-norm query guard passed everything. A norm of zero does
not imply a dot product of zero once components underflow: a query of
1e-200 against a doc of 1.0 scores 1.4e-200 without the guard, which
agrees with 0 to twelve decimals. Asserted exactly now.
* removing the clamp to [-1, 1] passed too. The self-cosine of
[1/3, 1/3, 8/3] computes to 1.0000000000000002 with sequential
arithmetic, and that also agrees with 1.0 to twelve decimals. Asserted
exactly, both ends of the clamp.
Two of my own test bugs on the way: `subset_criteria` is a LIST when the
criteria were a list and I assumed a map, and the precision guard multiplied
by 1e12 -- which is not exact in binary, so it failed on its own arithmetic
rather than on the data. It reads the decimal scale through BigDecimal now.
`embed_fn_from_agent` is deliberately not ported: it mean-pools the
checkpoint's own encoder, which needs hidden states, and the ONNX graph this
runtime loads exposes logits. The reference says a dedicated bi-encoder
shortlists better than that helper anyway.
455 tests pass.
47 findings across port fidelity, Java defects, test vacuity and the
generators. 30 were real and are fixed here; 4 turned out faithful to the
reference and were deliberately left alone; 11 were my own tests asserting a
bug or unable to fail at all.
THE TWO THAT MATTERED MOST WERE IN CI, AND ONE OF THEM MADE A CLAIM IN THIS
PR UNTRUE. The drift gate ran `--check --only python_json.json`: one of six
model-free families. lang_detect, lang_tables, presets, router and shortlist
were gated nowhere, so a change to laya/ could move the contract those five
record while every check stayed green -- the Java tests assert against the
committed copy, which is exactly the failure the fixtures exist to prevent. It
is a bare `--check` now, and it immediately caught real drift in three
families. And `--only` with a typo exited 0 printing "fixtures match laya", so
one hyphen for an underscore turned the whole gate into a no-op that asserted
the opposite; an unknown family name now exits 2, and the success path refuses
to report a match when it examined nothing.
Also removed four hardcoded copies of a developer's home directory as the
default checkpoint root. Besides being a leaked path, it made WHETHER a family
regenerates a property of whose machine ran the script, and wrote that path
into a committed skip marker.
Behavioural defects, worst first:
* a NaN cosine ranked FIRST where numpy's argsort puts it LAST. Double.compare
treats NaN as the largest double, so a descending comparator inverted it --
and that does not merely misreport a score, it changes which labels survive
the cut, so the model is asked a different question.
* a throwing Agent.close() during eviction aborted the build after the new
slot was published, stranding its lease: nothing could release it, so the
agent could never be retired or closed. An unreachable, unclosable native
session, and every waiter told the load failed when it had not.
* Lease.close() was documented idempotent but raced itself. Two threads
closing the same lease both passed the check, both decremented, and the
count hit zero while another lease was open -- closing an agent under a
caller still using it, the single failure the mechanism exists to prevent.
* eviction closed agents while holding the registry lock, stalling callers
asking for an already-resident checkpoint by as long as the close took.
Measured at 1.3 s. That is precisely what the fast path exists to avoid.
* attach() closed the very agent it is told the caller owns, when the router
already held that instance -- which is the case its own javadoc invites.
* String.trim() where the reference uses str.strip(), in normaliseName and
englishFromCode. Python's strip treats U+00A0, U+0085, U+2007 and U+202F as
whitespace and Java's trim does not, so "en" plus an ideographic space --
what a CJK input method produces -- routed to the MULTILINGUAL checkpoint
while the reason string still said the caller asked for English. The same
gap sat in Question's label handling and instruction check, where
UnicodeTables.strip already existed and simply was not used.
* the embedding cache had no dimension guard, so where the reference refuses
a changed embedder and keeps a clean cache, this accepted it and poisoned
the cache permanently; counted misses per distinct text where the reference
counts per occurrence, breaking hits+misses == texts looked up; and handed
out its own row arrays, so a caller's write corrupted every later lookup.
* a null question id aborted the whole route via Set.of().containsAll, where
the reference merely fails to match and carries on.
* percent0 spelled a non-finite as "NaN"/"Infinity" where CPython's %.0f
gives "nan"/"inf". Latent -- both call sites are finite by construction.
Left alone on purpose, because the reference does the same: maxLoaded is a
high-water mark; analyse() reports script and rates from the joined window
while taking the verdict from a leaf, which can emit "0% non-English letters;
not safe for the English checkpoint"; task rejects workflow names with a
message about models; cache misses are counted before the embed is attempted.
Each is now documented as deliberate rather than looking like an oversight.
On the test side: five @testfactory bodies generated zero assertions when
their fixture key emptied, and the suite exited 0 -- they have floors now,
each naming the row that has to exist. Two score loops had no length
assertion. A hostile-case test asserted the throw on the first question in a
spec rather than the one missing its instruction. One assertEquals had its
arguments reversed. And four regression tests now cover the lifecycle defects
above, including the close race and the stranded lease -- both of which my own
new tests caught on the first run, before the fixes were complete.
480 tests pass. Every generator gate is clean and regenerating twice gives
identical bytes.
Two more adversarial passes, this time over the code that merged in NandhaKishorM#927 and had only ever had mutation testing and parity fixtures. 25 findings; the release-blocking ones first, because any of them could have done damage that cannot be undone. RELEASE AND CI * release-java.yml on a manual run from a BRANCH published to Maven Central with the branch name as the version. GITHUB_REF_NAME is the branch there, so the version became `main`; the snapshot guard only matched `*-SNAPSHOT` and let it through; and the build job's version check was guarded by a tag condition the publish job did not share, so the two jobs disagreed about what was being released and the artifact published was never the one the tests ran against. Central is append-only: `:main` could never have been replaced. It now refuses to publish from anything but a tag. * the release gate's test floor was 80 against a 484-test model-free suite, so 83% of the tests could vanish and the publish still went ahead. Now 440, matching the build lane. * check_test_results.py compared the floor against JUnit's `tests` attribute, which COUNTS SKIPPED TESTS. A suite in which every single test aborted satisfied a floor of 440 and both --allow-aborted lanes -- the build gate and the Central release gate -- went green with nothing executed. That is the exact failure the script's docstring says it exists to catch. It counts executed tests now, and an all-skipped suite fails. * the empty `laya-java-client` module was in the Gradle build with a full MavenPublication, so a release would have pushed a zero-class jar to Central with a POM promising an HTTP client the README itself lists as not implemented. It joins the build when it has code. * publishing required two of the four secrets. Without the signing passphrase the build calls useInMemoryPgpKeys(key, null) and uploads UNSIGNED artifacts, which Central rejects only after the upload. THE SPLIT GRAPH LAYOUT WAS DEAD. `encoder.onnx` + `head.onnx` is a documented, advertised layout, and the head declares `qtype` as [B, 1] while this passed a rank-1 tensor -- so ORT refused every graph the only exporter for that layout produces, and the error surfaced as a per-batch backend failure that reads like a bad model rather than a bad client. laya-ts passes [n, 1] for the same reason. OTHER RUNTIME DEFECTS * a temperature written as a JSON string was discarded and silently defaulted to 1.0, where the reference parses it. Measured: the published answer_confidence -- the number callers gate on -- was overstated by 19 points with no warning, in a class whose whole stated purpose is that these are read from the checkpoint and never defaulted. * a temperature of the wrong TYPE (a number, a string, a mapping) loaded cleanly at 1.0 where the reference refuses the checkpoint. The wrong LENGTH was already refused; only the type slipped through. * a logits output of the wrong rank or dtype escaped as a raw ClassCastException naming neither the graph nor the output, through a method whose javadoc promises the backend never leaks -- so a caller's catch (InferenceException) did not catch it. An fp16 or fp64 re-export reaches it. * run() never consulted `closed`, so the guard isClosed()'s own javadoc promises ("a caller gets an API error, not a backend one") did not exist, and a use-after-close produced exactly the error class the API says it never emits. * the action softmax padded a one-column head up to two columns, turning a probability of 1.0 into a sigmoid -- a plausible-looking number for a value the model never produced, in the field callers gate escalation on -- and published NaN for a zero-column row. It uses the graph's own width. * round4 lost the sign of a negative zero, so the exact-parity claim did not hold and a record's equals() would have reported two equal answers as different. THE JSON LAYER, which parses every fixture and every tokenizer.json * no depth limit in either direction: deeply nested input threw StackOverflowError -- an Error, which a server's catch (Exception) does not contain -- where CPython raises a catchable RecursionError. The identity-set fix had closed only the cyclic case. Both now have one. * an integer literal was converted with a QUADRATIC BigDecimal call and no digit cap: a 1 MB literal cost 21 seconds of a request thread, from one field of a checkpoint downloaded off the hub. Capped at CPython's 4,300. * a backslash-u escape accepted fullwidth and Arabic-Indic digits as hex, because Character.digit is Unicode-aware, so two byte-different documents parsed to the same value. ASCII only now. 481 tests pass, every generator gate is clean, and one test that had been asserting the temperature defect was corrected rather than the fix weakened.
…s Unicode, not the reference's
The only live defect of the whole review: it fires on the shipped english
checkpoint, with no config change, and the same jar gave different ids on a
different JDK.
The GPT-2 pre-tokenizer pattern is
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+
and `\p{L}`/`\p{N}` in java.util.regex follow the JDK's Unicode version.
Corretto 17 ships Unicode 13.0; the reference tokenizer classifies Unicode
15.1 and 16.0 additions as letters. They disagree on 9,917 code points -- CJK
Ext-I, Egyptian Hieroglyphs Ext-A, Arabic Extended-B, Vithkuqi, Kawi, Tangsa,
Garay, Todhri, Tulu-Tigalari and others, plus several scripts' digits.
For every one of those the seven contraction tokens could never form: when
` ?\p{L}+` fails, ` ?[^\s\p{L}\p{N}]+` swallows the apostrophe into the same
piece, so `'s` `'t` `'re` `'ve` `'m` `'ll` `'d` are never produced. Measured:
99,170 of 119,004 probe inputs diverged, including the ordinary frame "The X's
value". Three Unicode versions are in play and no two agree -- the JDK's
13.0, CPython's 15.0 (which this repo's other tables record), and the
reference's -- so the tables already here would not have fixed it either.
The classes are therefore recorded FROM THE REFERENCE and compiled in, which
is what every other table in this port does. They cannot be read as a
property, so the probe is behavioural and follows from the pattern itself: a
code point is a letter if "<cp>'s" leaves "'s" as its own piece, and a number
if "1<cp>" stays one piece. A full sweep of all 1,114,112 code points takes
about five seconds, and `--check` gates the committed copy in the parity lane,
which is where a checkpoint is available.
The pattern is now a scanner over those tables rather than a regex, in the
same style and for the same reason as the language detector's four scanners.
The alternation order is the specification and is preserved, including the
`\s+(?!\S)` backtrack that leaves a whitespace run's last space to the
following word -- which is what produces "Ġthere" rather than a lone
whitespace piece. Whitespace stays `Unicode.isWhiteSpace`, which already
matches Rust's `char::is_whitespace` on every code point; that is what
UNICODE_CHARACTER_CLASS was buying and the only part of the flag that worked.
Verified against the reference on both shipped checkpoints: 216 targeted cases
(all seven contractions against the code points that used to diverge) and a
34,395-case sweep striding every plane in three frames -- 0 divergent, where
the regex diverged on 99,170 of 119,004.
481 tests pass.
The reference's rule is `truncate_left = isinstance(state, list)`, and this
passed a literal `false`. A conversation is serialised newest-last, so an
oversized history was cut from the right: the model was shown the OPENING of
the conversation and answered about that, while the current turn -- the thing
the question is about -- was the part thrown away.
Measured on a 120-turn history: the reference's window started mid-message-95
and ended at turn 119; this one started at turn 0 and ended at turn 24. End to
end on the real fused graph, one option's probability moved by 3.8x (0.0364 to
0.0096) and the score moved 2.0685 to 2.0265. And `usage` was byte-identical
between the two -- same input_tokens, same state_tokens_dropped, same
truncated flag, same truncated_questions -- so nothing a caller could read
revealed it. Left-truncation is the entire reason that flag exists, and the
only production caller never set it.
Three smaller parity defects alongside it, all found the same way:
* a null state was answered as a decision about the literal text "null",
byte-identically to passing the string, at full confidence. Both Python
backends raise `TypeError("state must not be None; ...")` and the
reference's own comment gives that serialisation as the REASON to refuse.
The comment here claimed the opposite -- that a null state is legitimate
"because Python serialises None to the text null" -- which would have
stopped the next reader from fixing it. So would the test, which was
titled "a null state is answered, as Python answers it".
* an empty questions map threw, where the reference returns empty answers
and zero usage per state with no tokenization and no forward pass. That
meant the two runtimes could not be swapped under a question set derived
from a filter, where an empty schema or a fully disabled rule set is a
well-formed request with a well-formed empty answer. The test asserted the
throw.
* a checkpoint naming no pad_token silently fell back to the SEP id, where
the reference fails the request. The graph masks padding so the numbers
did not move; the divergence was that a mis-exported checkpoint failed
loudly on one runtime and answered quietly on the other.
Two of the three had a test pinning the wrong behaviour, and one of those
asserted a false claim about the reference in its own display name. Both are
corrected rather than deleted, and the left-truncation fix has a regression
test that decodes the surviving window and asserts the newest turn is in it
and the oldest is not.
Worth recording what the same review found CLEAN, because it is the part that
was most at risk: the head budget and option truncation over 1,483 cases
compared id-for-id (including 506 re-capped, 400 option-collapse, 72
marker-dropping and 337 truncating cases); row attribution across 37 uneven
states x 4 questions x 10 batch and sort configurations, 370 of 370 identical;
the collator's padding and masks over 192 collations, element for element; and
every usage field including which question is blamed for a truncation. Those
are the places an off-by-one silently returns another state's answer, and they
hold.
482 tests pass.
The README still described the SDK as prediction-only. Add the four sections for the features this branch adds, and correct the status list so it no longer claims the router and detection are missing. The "lower-level pieces" snippet was also missing three imports it uses (AgentConfig, Tokenizer, Path), so it could not be pasted as written. Every java block in the file now typechecks against the compiled classes.
…question
The probe behind PreTokenizerTables asked whether "<cp>'s" leaves "'s" as its
own piece. That is not a test of class membership: it is true of every code
point that fails to ABSORB the apostrophe, which includes every whitespace
code point and every digit. 1,935 code points were recorded as letters that
are not -- all 1,911 digits and the 24 non-space whitespace characters -- and
LETTER and NUMBER overlapped on all 1,911.
Because ` ?\p{L}+` is tried before `\s+`, a letter run then swallowed inner
whitespace: "a b\t\tc\n\nd\r\ne f" encoded to 11 ids where the reference
gives 14, the tabs and newlines merging into the surrounding word instead of
becoming their own pieces. "abc123" collapsed to one piece where the reference
gives two. Both fire on the shipped english checkpoint with no config change.
The multilingual checkpoint is unaffected -- its Metaspace pre-tokenizer does
not use these classes -- which is why only one corpus entry failed.
The probe is now anchored by a character of the class being tested: a code
point is a letter if "a<cp>" stays ONE piece and a number if "1<cp>" stays
one. The only way the first match can reach past the anchor is the alternative
under test consuming the code point, so one piece means membership and two
means non-membership, with no third reading. The misleading comment claiming
the old judge was written so "only a genuine class membership answers true"
is replaced by what makes the new one sound.
Worth being plain about how this got in: the sweep quoted in 0e1f1b1 was 216
targeted contraction cases and a 34,395-case stride sweep in three frames,
none of which put whitespace or a digit BETWEEN two letters -- the one shape
that makes an over-inclusive letter class observable. The corpus fixture
caught it, and only with a checkpoint attached. The sweep now covers 115,699
cases including whitespace and digit frames in twelve positions: 0 divergent
on both shipped checkpoints, where the old table diverged on 32 of them on the
english checkpoint and on none on multilingual.
PreTokenizerTablesTest pins it model-free, because regenerating cannot catch a
bad probe -- the generator agrees with itself whatever it asks. It asserts the
properties the pattern's alternatives imply rather than the table's contents:
no whitespace code point is in either class, the two classes are disjoint, an
inner whitespace run is cut exactly as the reference cuts it, and a digit run
stays its own piece beside a word. Four of the five fail against the old
table. The fifth guards the Unicode skew 0e1f1b1 fixed, over 751 code points
across five blocks that are letters to the reference and not to Corretto 17.
487 model-free tests pass, 634 with the checkpoints and the fused graph
attached, and every generator gate is clean.
The java lane had two jobs where the .NET lane has five, and neither of the
missing three was a cosmetic gap.
A JDK MATRIX, which is the one this port actually needed. `\p{L}` and `\p{N}`
in java.util.regex follow the JDK's own Unicode version, and that is how the
pre-tokenizer came to return different token ids from the same jar on
different JDKs. Measured: `Character.isLetter` answers 0 of 751 on Corretto 17
(Unicode 13.0) and 751 of 751 on Corretto 24 (Unicode 16.0), across the five
blocks this port has to classify.
A lane that merely installs another JDK would not have caught it, because the
toolchain pins the COMPILER to 17: Gradle would compile and test on 17 whatever
setup-java installed, and the cell would go green having tested the same thing
twice. So `-PtestJavaVersion` moves the test JVM only; `options.release` keeps
the bytecode at 17, verified as class file major 61 while the tests ran on
24.0.2. Two guards stop it being vacuous. `launcherFor` with auto-download off
fails the build outright on a JDK the runner does not have -- "Cannot find a
Java installation on your machine matching: {languageVersion=19}" -- instead of
falling back to 17; and TestJvmVersionTest asserts the running JVM from inside
the suite, because Gradle writes `<properties/>` EMPTY into the JUnit XML, so
neither the report nor any later step can tell 17 from 24. Forcing the launcher
to 17 while the lane asks for 24 fails that test.
The model-free suite is the right scope for the matrix: the JDK-sensitive tests
are exactly the ones needing no model -- the character tables behind a sha256
over all 1,114,112 code points, the White_Space predicate, the pre-tokenizer's
letter and number classes, CPython's float repr and the rounding rules. All are
compiled in FROM the reference so the host JDK's Unicode cannot move them, and
this is what measures that. 635 tests pass on 17 and 24 alike, 0 skipped.
THE PARITY LANE IS NOW TWO CELLS, split by whether the cell needs the traced
graph. Only one of the four checkpoint-backed test classes does; the other
three need `tokenizer.json` and `rl_agent_config.json`, which are kilobytes.
Measured: the tokenizer cell is 183 tests in 8 seconds against 3 fixture
families regenerated in 13, and it was waiting on a gigabyte-scale export it
never reads. That export is the whole runtime of this lane, and it is also
where the pre-tokenizer bug was caught -- a check that cheap should not take 75
minutes to report.
NOT split per checkpoint, which is where the .NET lane splits, and the reason
is worth recording so nobody "fixes" it later. The fixtures are not
per-checkpoint documents: tokenizer_ids.json, sequences.json and decode.json
each carry BOTH checkpoints' sections in one file, so a job holding one of them
regenerates none of the three and --strict fails by construction. And the Java
tests assert that both checkpoints' cases are present -- a floor that exists
because a @testfactory whose fixture key emptied used to generate zero
assertions and still exit 0. Splitting per checkpoint would mean deleting that
floor to buy nothing, because there is no export to parallelise away. Verified
by hand: with one checkpoint present the suite reports a FAILED container, not
a skip.
Smaller things the split turned up:
* a blank env var is not an absent one. A matrix cell that owns no graph
writes `LAYA_ONNX_GRAPH: ''`, because an expression yielding the empty
string still DEFINES the variable, and the build passed it straight
through -- so the graph-backed factories would have failed on an
unopenable path where they are written to abort by assumption. Blank now
counts as absent, and with it the suite reports identically either way:
628 tests, 3 skipped, both for unset and for empty.
* the floors are per cell now, 175 and 9 against 183 and 9 measured. Dropping
three of the tokenizer cell's four classes leaves Gradle reporting BUILD
SUCCESSFUL on 5 tests, and the floor is the only thing that catches it.
* `--tests` patterns are fully-qualified class names, not `*Glob`. An
unquoted `*Foo` interpolated into a `run:` line is a bash glob that
survives only while nothing in the working directory matches it.
* the hyphenated matrix key is read as `matrix['min-tests']`; by dot access
an expression parser can read the hyphen as subtraction.
* the new job runs check_test_results.py, so it sets Python up like the other
two rather than trusting the runner image to expose `python`.
* setup-java's variable suffix follows `runner.arch` rather than a hardcoded
`_X64`.
The lane stays advisory, like the .NET one and for the same reason: the Python
package is the product, and a Python-side improvement must not be blocked on a
port catching up.
Six findings against the lanes added in the previous commit. Three of them made
a cell green while measuring the wrong thing, which is the failure a CI change
is most able to hide.
THE JDK FLAGS WERE SILENTLY IGNORED. Gradle does not read
`org.gradle.java.installations.*` from `-D`; it must be `-P` or
gradle.properties. Measured on this build with the `javaToolchains` report:
`auto-detect=false` passed as `-D` left all 8 locally detected toolchains in
place, and as `-P` dropped it to 3. So `fromEnv` never registered setup-java's
tool-cache installs and `auto-download=false` never applied -- and the comment
claiming those two made a misconfiguration loud described something that was
not running. The cell passed anyway, by coincidence: setup-java makes the LAST
`java-version` the JAVA_HOME, so Gradle's "current JVM" supplier happened to
supply the right launcher while the image's own JDK 17 served the compiler.
That is precisely the fragility naming the variables was supposed to remove.
Now `-P`, with `auto-detect=false` added so the only installations are the two
named. Verified both directions: asking for 24 with auto-detect off and no
`fromEnv` exits 1 with "Cannot find a Java installation on your machine
matching: {languageVersion=24}"; adding `fromEnv` exits 0.
`--allow-aborted` FORGAVE THE ONE TEST THAT PROVES THE JDK LANE IS NOT VACUOUS.
TestJvmVersionTest is `@EnabledIfSystemProperty`, and the build sets that
property only when `-PtestJavaVersion` is present. If the flag ever stops
reaching the build -- renamed on one side, lost from a folded `run:` block,
overridden in gradle.properties -- then no launcher override happens, both
cells run on the compile JDK, and the single test that would notice is SKIPPED
rather than failed. This lane needs `--allow-aborted`, because the
checkpoint-backed factories abort here, and that flag permits exactly that
skip. Measured with the flag dropped: Gradle exits 0, the floor exits 0 on 488
tests, and the lane reports two JDKs having run one of them twice.
`check_test_results.py` grew `--require-passed`, which demands a substring
match against a test that actually PASSED rather than merely registered; with
it, the same run exits 1 and names why ("it was skipped").
THE EXPORTED-GRAPH CACHE KEY DID NOT INCLUDE THE PINNED REVISION. HF_REVISION
lives in prepare_checkpoint.py, which was not in the key's `hashFiles`, and the
script short-circuits on an existing graph. So bumping the revision changed
nothing the key could see: the graph traced from the PREVIOUS revision was
restored and reused, `predict.json` was re-recorded from that same stale graph,
and the two sides agreed. Nine tests pass, cell green, having never measured
the pinned checkpoint -- with the informational drift step showing
`predict.json` unchanged, which reads as reassurance. The sibling lane does not
have this bug; its key hashes the file its own revision lives in.
Fixed at the source rather than only in the key: prepare_checkpoint.py now
stamps the revision beside what it produces and reuses a checkpoint or a graph
only when the stamp matches, so a restored cache from before a bump is
re-downloaded and re-exported instead of trusted. The stamp also records
whether weights were fetched, because a weightless checkpoint cannot be traced
and must not satisfy a request that needs one. The key additionally carries the
cell name and prepare_checkpoint.py as defence in depth.
THE HUGGING FACE CACHE WAS A PERMANENT NO-OP, and is removed rather than left
looking like a cost that was being avoided. prepare_checkpoint.py passes
`local_dir` to `snapshot_download`, which writes the files there and not into
the hub cache. Measured against the pinned huggingface_hub: with a revision
pinned, as here, `~/.cache/huggingface/hub` gets nothing at all; without one it
gets a `refs/main` pointer and zero blobs. The step therefore always missed on
restore and saved nothing, every run paid the full download, and its key also
lacked the cell name -- so the moment it was made to work, the two cells would
have raced for one key while holding different content and the weight-bearing
cell could have restored a weightless tree. What making it real would take is
recorded where the step was.
Two smaller ones:
* the test floors were 440 against suites that execute 487 and 488, so a
third of either could vanish unnoticed -- and that floor is the only thing
between a @testfactory that stopped registering cases and a green cell.
Raised to 480. NOT to the ~620 a full run reaches: that is a
checkpoint-bearing total, and a model-free lane would fail it on the first
run.
* `matrix['min-tests']` is back to `matrix.min-tests`. The bracket form was
defensive about a hyphen that needs no defending -- actionlint types the
key and lints the dot form clean, and the .NET lane already spells it that
way. Two lanes should not spell the same thing two ways.
All five lanes verified against their exact commands: 488 executed in the JDK
cell with TestJvmVersionTest passing, 487 in the build lane, 183 in the
tokenizer cell, 9 in the predict cell. actionlint with shellcheck is clean on
every workflow in the repository except docker.yml, which is untouched here and
has a pre-existing SC1007.
Groundwork for the email port, as its own commit because the tables are gated
exhaustively and the port is not written yet.
`laya.email` reads two character properties this port did not record, and both
were nearly implemented by reusing a table that already existed. Both reuses
would have been wrong, on large and specific sets, so the numbers are asserted
rather than left to a comment.
MARK is the Mn/Mc/Me categories -- `unicodedata.category(ch).startswith("M")`
in `_drop_marks`, `\p{M}` in the TS port. COMBINING is not a substitute: it is
the canonical combining CLASS, and 1,528 code points are category M with a
class of zero -- U+034F COMBINING GRAPHEME JOINER, the Cyrillic Me numerals,
the Thaana vowels. Marks are dropped before a sign-off's tail is matched
structurally, so using the class would leave those in and a name carrying one
would stop reading as a name.
INITIAL is the Lu/Lt/Lo categories, the test `_is_english_signoff` applies to
each token's first letter. UPPER is not a substitute either, and not merely
imprecise: `str.isupper()` is the Uppercase property, true of 1,951 code points
against these 133,474, and the two sets are not nested in either direction.
131,643 initials are not uppercase -- every caseless script, so "Obrigado,
山田" would stop being a sign-off -- and 120 uppercase code points are not
initials, the circled and squared letters of category So, so "Thanks, Ⓐ" would
start being one.
Recorded from CPython 15.0.0 like every other table here, with an exhaustive
sha256 over all 1,114,112 code points in the lang_detect fixture, and a third
test pinning the three counts above plus the five code points that make the
difference observable. Delegating isMark to COMBINING or isInitial to UPPER
fails four assertions each.
638 tests pass, and every generator gate is clean.
The cleaner and the state builder, matching the reference byte for byte on 93
recorded cases and on a 40,000-case differential sweep: 0 divergent.
WHAT HAD TO BE TRANSLATED RATHER THAN TRANSCRIBED. Four of Python's regex
constructs mean something else in java.util.regex, and three of them are the
kind that are right on the cases you thought of:
* `\s`. Python's is 29 code points, exactly `str.isspace()` -- verified over
all of Unicode. Java's is 6 by default and 25 under
UNICODE_CHARACTER_CLASS, which is the White_Space property and therefore
misses U+001C to U+001F; those four are Cc, and Python calls them space.
* `\d`. Python's is Nd, 680 code points. Java's is ASCII by default and,
under UNICODE_CHARACTER_CLASS, follows THE JDK'S Unicode version -- the
defect this port shipped in its pre-tokenizer, where the same jar gave
different token ids on different JDKs. A date written in Arabic-Indic or
Devanagari digits is a date, and an ASCII-only port keeps the quoted
history instead.
* `.` and `$`. Python's `.` excludes only `\n`; Java's excludes every line
terminator, U+0085 and U+2028 among them, and its `$` looks for any of
them. Every pattern carries UNIX_LINES so both mean what Python means.
* `\b`. Below.
`\s`, `\S` and `\d` are therefore built FROM the recorded tables rather than
written out, so they and the rest of the port read one Unicode and a table
change moves both together.
THE SIGN-OFF RULE is a scanner, as `laya.lang`'s four are and for the same
reason: its classes are Python's. `[^\W\d_]` opens a token and `\w` continues
it, and Java's `\w` is a different set -- it admits the combining marks the
reference has just dropped and the join controls that must keep separating
tokens. Each token's first letter must be Lu, Lt or Lo, which is what the two
new tables are for: `str.isupper()` would lose every caseless script
("Regards, 山田" is a sign-off) and gain the circled letters of category So
("Thanks, Ⓐ" is not).
`\b` COST 220 TIMES WHAT IT WAS WORTH, and the way out is the interesting part.
A word boundary is defined in terms of `\w`, so the exact translation is a
lookaround over Python's class -- which inlines 10,622 characters at 26
boundary sites, makes a 277,309-character pattern, and runs at 1,742 us per
paragraph against 7.8 us for Java's own `\b`. It was also the whole cost of
cleaning a message: 2.2 ms per case, against Python's 0.026 ms.
The two forms agreed on all 20,000 paragraphs of the sweep, and that is exactly
why the lookaround was not kept as the safe option: agreement on a corpus is
not equivalence, and Java's `\w` is the JDK's. So the boundary is still decided
by the recorded table, but outside the pattern -- `boundaryView` rewrites the
input so that Java's `\w` and Python's agree on every character of it, and
Java's fast `\b` then finds Python's boundaries. Both directions of the
disagreement are real and both are exercised: a mark or join control is Java's
word character and not Python's, so it becomes U+0000; U+00BD and the other
other-numbers are Python's and not Java's, so they become '0'. Both substitutes
are one character wide, are matched by the `[^.]` windows the patterns measure,
and appear in no literal of any pattern here -- which is asserted, not argued,
because it is the whole reason a substitution is allowed to be invisible. The
corpus rewrites 12,583 of 40,000 bodies, so neither branch is dead.
Three more measurements, each made before the change it justifies:
* a necessary-condition filter in front of the disclaimer pattern, since
every branch of it requires one of 24 literals. 2.2 ms to 0.99 ms. It is a
filter and never a decision, so a false positive costs only time -- and it
is matched with the pattern's own flags rather than by lowercasing, because
`re.I` folds more than lowercasing does and a filter that missed U+017F
would reject a disclaimer the pattern accepts.
* `javaWord` spelled out from the definition the JDK documents for `\w`,
rather than asked of a matcher per character. Compared against that
matcher over all 1,114,112 code points: identical.
* the character classes built in one walk of Unicode instead of three, and
the unused word class removed from the holder. First call into the class:
160 ms to 52 ms. The classes are byte-identical to the three-walk version,
which is what makes the refactor a refactor.
End to end: 0.029 ms per case against the reference's 0.026, warm and best of
three on the same 4,000 bodies -- from 85 times slower to parity, with the
exactness unchanged at every step. The patterns are built on first use, so a
caller who only ever opens an Agent pays none of it.
WHAT THE CORPUS IS. 93 cases, and the near-misses outnumber the hits, because
the module argues in four separate places that a cleaner which is too eager
deletes the sender's request. `From: my side the integration works` is prose,
`Em resposta ao que você escreveu:` carries no date, `De: 10/09 a 15/09` is a
leave request, `Thanks for the quick reply.` is a sentence, `Obrigado pelo
retorno, mas ...` is a request, and `Is this confidential?` is a question.
Ten mutants were written against the port and SIX SURVIVED the first corpus --
the combining-class table, the mark-without-a-base rule, the 40-character
sign-off limit, the three-token tail, the attribution head, UNIX_LINES and both
space classes were all invisible to it. Eight cases were added to kill them, a
non-ASCII-digit case for a seventh, and the thirteen mutants are now all killed.
That is the second time in this branch that a green fixture meant nothing until
a mutant proved it could fail.
742 tests pass, identically on JDK 17 and JDK 24, every generator gate is
clean, regenerating email.json twice gives identical bytes, and all ten of the
README's java blocks compile against the built classes.
…rt added 594 model-free tests execute now, against floors of 585. They were 480, which was the same slack the review of these lanes flagged when the suite was 487 -- a floor that drifts behind its suite stops being a gate, and this one is the only thing between a @testfactory that quietly stopped registering cases and a green cell. Measured, not computed: `./gradlew test` with no checkpoints reports 599 found and 5 skipped, and the five skips are the three checkpoint-backed factories plus the JDK-version guard, each of which is a deliberate abort in this lane.
The 40,000-case sweep that cleared the port had NO astral character in it. A
20,000-case corpus of them -- marks, letter- and other-numbers, mathematical
letters, CJK extensions, emoji, at every offset of a disclaimer and at every
width around the sign-off limit -- diverged on 4,609. The units the reference
counts are code points and the units Java reaches for are chars, and that is
three of the four; the fourth is mine.
THE BOUNDARY VIEW WAS RECONCILING AGAINST A CLASS NOTHING CONSULTED. The
previous commit replaced the exact `\b` lookaround with Java's `\b` applied to
a string rewritten so that Java's word class and Python's agree on it. Java's
`\b` only reads the Unicode word class when UNICODE_CHARACTER_CLASS is set, and
it was not, so `\b` was reading `[a-zA-Z0-9_]` while the view was built against
the Unicode one. The symptom is a false positive, which is the direction that
costs a user something: "Esta mensagem" followed by a letter-number has no word
boundary to Python, an ASCII `\b` finds one anyway, and a sentence the reference
keeps was deleted whole. The flag is set on that one pattern now; it has no
`\s`, `\d` or `\w` of its own -- those are spelled out from the tables -- so the
flag reaches nothing else.
The other three are counting:
* the final cut. `text[:max_chars]` is code points; `substring(0, maxChars)`
is chars and splits a surrogate pair, returning half a character the
reference cannot produce. 4,641 of the 4,609 divergent cases are this one.
* the pre-truncation bound, `max_chars * 4`, the same way. It decides whether
a disclaimer is inside the window at all, so cutting it short lets a
truncated footer that no longer matches survive into the model's input. The
final cut hides this for most inputs, which is why the case that pins it is
specific: three astral characters and a budget of 11.
* the signature width. `len(line.strip()) <= 40` counts code points, so a
40-code-point closing is 41 chars wide with one astral letter in it and the
signature stays.
Nine cases are added to the corpus, one per defect plus the near-misses that
keep each fix from being "never match": a disclaimer whose astral character is
somewhere harmless still matches, and a sign-off one code point over the limit
still survives. All four mutants die against them, and the first version of the
pre-truncation case did NOT kill its mutant -- the final cut masked it -- so the
case in the corpus is the one found by searching for a discriminating
(padding, budget) pair rather than the one I wrote first.
Both sweeps are clean: 0 of 20,000 astral and 0 of 40,000 on the original
corpus, which is the regression check for fixes aimed at the first. 751 tests
pass, identically on JDK 17 and JDK 24.
Worth saying plainly, because it is the second time on this branch: the port
had already been through 93 recorded cases, a 40,000-case differential sweep
and thirteen mutants, and all of that passed while four defects sat in it. A
corpus only tests what it contains, and mine contained no character above
U+FFFF.
Twenty-two findings. One is a runtime defect, four are tests that could not
fail, and the rest are claims in comments that were not true.
A HOSTILE LINE COST 4.8 SECONDS. `ATTRIBUTION_TAIL` is
`^.{0,120}\S@\S+\s+(wrote|...)\s*:\s*$`, and `.{0,120}` offers 121 starting
points; at each one a greedy `\S+` ran to the end of the line and gave every
character back, because the `\s+` after it can never match where a `\S` was
just surrendered -- the classes are complements. 121 full scans of the line.
Measured on a 4,000,000-character line: 55 ms at the default budget and 4.8 s
at a `maxChars` of 300,000, against 0.018 ms for an ordinary message -- and
`emailState`'s own javadoc tells callers that raising `maxChars` is worth it.
Two changes: `\S++` possessive, which forbids backtracking that could never
have succeeded, and a necessary condition in front of the pattern, since a tail
that matches is a line whose rstrip ends with a colon. 55 ms to 9 ms, 4.8 s to
66 ms, both sweeps still clean.
FOUR TESTS COULD NOT FAIL, and the filter they were guarding had five dead
literals because of it.
* the disclaimer filter's necessity test checked thirteen examples against
the WHOLE filter. Every example was satisfied by some literal, so a literal
that only one branch needs was invisible: `sigilos`, `exclusivamente`,
`uniquement`, `recebido` and `receber` could each be deleted with the suite
green, and "Esta mensagem e sigilosa." then reached the model. The filter is
now twelve literals, each the shortest form covering its branch, and the
test gives each one a disclaimer whose ONLY hint literal is that one and
asserts that removing it stops the filter matching. All five previously
deletable literals now fail. (`privilegiad` really was redundant --
`privil[eé]gi` covers it -- so it is gone rather than tested.)
* three of the pattern's twelve alternations were never the sole matcher, so
breaking them killed nothing. The per-literal table covers each.
* the @testfactory floors were counts well under the corpus: 70 against 86,
5 against 9. Trimmed to those floors the suite stayed green while losing
every astral case, and the three mutants the full corpus kills -- the
code-point width, the surrogate-safe cut, the `\b` flag -- all survived.
The corpus is now checked by NAME: thirty cases are listed as load-bearing,
and the sizes are exact equalities to be changed knowingly.
* the signature window `max(1, min(int(len * 0.6), len - 8))` was entirely
unpinned: the ratio to 0.55 and 0.9, the offset to 9 and 2, the floor to 2,
all green. The device-footer width was mutable from 60 to 41 and to 200.
The body-side strip swapped for `String.trim()` was green, and then a body
wrapped in U+00A0 kept its space -- the same Python-vs-Java strip gap the
subject side already pinned. Nine cases added, the five window ones found
by searching for the smallest input each mutant disagrees on rather than
derived by hand. All eight mutants now die.
Two of my own fixes were themselves vacuous, which is the part worth recording.
The NUL check scanned the pattern SOURCE for a raw NUL -- a thing no pattern
can hold, since every class member is written as an escape -- while the two
forms a NUL literal actually takes read as ordinary text to `indexOf`. It is
behavioural now: no pattern may match a lone NUL. And the categories overload
had no test; the one I wrote compared `toString()` on maps of `Question`, which
has none, so it compared identity hashes and was true however the method
behaved. It compares labels now. Both mutants die.
Claims corrected rather than left standing: the filter is not "40 times
cheaper" (it is at parity with the pattern on a paragraph holding no literal,
and worth 1.16x on the corpus -- the 40x was true only while `\b` was a
lookaround); DISCLAIMER has twelve alternations and 26 boundary sites, not
"twenty alternations each carrying two"; the walk is 15.8 ms and the first call
52-57 ms, not the two contradictory figures that were there; `'0'` is safe
because DISCLAIMER holds no digit literal, not because no pattern does -- two
others do. The surrogate figure is now attributed to a one-off sweep that is
not in the tree, and points at the four committed cases that hold the line.
Also: `boundaryView`'s invariant is asserted in code points, not chars -- the
char assertion was false for an astral disagreement and passed only because
every case in the list was in the BMP; an empty character class now throws
something that names the table instead of arriving as `NoClassDefFoundError` on
the second call; a dead `JAVA_WORD` pattern is gone; `isMark` and `isInitial`
use the ASCII fast path the generator was already emitting masks for; the
`@return` says what an extra named `subject` does; and four raw NUL characters
that had ended up in the test source are escapes again.
763 tests pass, identically on JDK 17 and JDK 24. Both sweeps are clean -- 0 of
40,000 and 0 of 20,000 -- every generator gate is clean, email.json regenerates
to identical bytes, and actionlint with shellcheck is clean on the workflow.
What the review found CLEAN is worth recording too, since it is where I
expected the bugs: surrogate handling throughout, integer overflow at
`Integer.MAX_VALUE`, `subList` on an empty body, thread safety of the lazy
holder, mutability of the returned question map, and every pattern but
ATTRIBUTION_TAIL on a 192,000-character hostile line.
The port-fidelity review of the email module, run against the reference line by line. Two findings; the first is a divergence on a narrow input and the second is the kind that is silent. A NEGATIVE max_chars IS NOT A ZERO max_chars. Python slices twice -- once at `max_chars * 4` and once at `max_chars` -- and a negative index drops the LAST |n| code points rather than returning nothing. For a budget of -5 the reference answers "I cannot log in a"; this port answered the empty string. 17 of 171 body-and-budget combinations differed and every one of them was negative. The cut now implements Python's slice for either sign, in code points, and the `len(text) > max_chars * 4` guard goes with it: it is redundant once the slice is faithful, and always true for a negative bound anyway. The test that was here asserted the empty string as correct -- "a non-positive budget returns nothing rather than throwing" -- so it is corrected rather than the fix weakened. That is the third test on this branch found asserting the behaviour it was supposed to be checking. Its replacement reads its expectations off the reference, because the first version I wrote guessed "a mess" for a budget of -3 and was wrong: both slices apply, so a body only survives a budget of -n if it is longer than 5n code points, and "a message" at -3 is empty after the FIRST slice. AN `extra` COULD SHADOW A PARAMETER AND NOTHING SAID SO. In the reference none of these can become a state field: `subject`, `body` and `max_chars` raise a TypeError, and `sender` and `clean` bind the parameter instead. A Java Map has no such protection, so every one of them was being put into the state. `body` is the one that bites -- the state is model input, so the value lands there UNCLEANED, with no quoted history removed, no signature cut and no budget applied -- and a caller translating `email_state(**payload)` got silence where the reference gives an error. All five are refused now, rather than emulating the two that bind: a Java caller has explicit parameters for those, so an entry naming one is ambiguous rather than meaningful, and erroring is the direction that cannot ship the wrong thing to a model. `from` is NOT reserved, because the reference lets it through, and the test pins that it replaces the sender in place rather than being appended. 115 recorded cases match, 768 tests pass identically on JDK 17 and JDK 24, both sweeps are clean -- 0 of 40,000 and 0 of 20,000 -- and both new mutants die. What that review found equivalent is worth recording, since it was looking for exactly the things this port gets wrong elsewhere: the three character classes over all 1,114,112 code points INCLUDING once compiled under the case flags, which widen ranges; `boundaryView`'s soundness over every code point; the sign-off tail scanner against the regex over 145,039 cases; `DEVICE_FOOTER` and `SIGNOFF_ROMANCE` over their full grammars, 91,564 cases; DISCLAIMER against the reference over 20,822 cases with word-class-stressing code points spliced in at every boundary; case folding including U+017F and U+212A; `.` and `$` under UNIX_LINES; and the whole cleaner over 14,720 differential cases. No integer overflow at Integer.MAX_VALUE. Nothing there to fix.
…fore they ship
The three deliverables the PR promised and had not delivered. `laya-java` was
the only one of the three ports with no samples at all; .NET ships three and TS
ships two.
MODELS.md: what `Agent.open` actually takes -- two directories that already
exist -- and the one script that makes both. It documents the `--no-graph` path
first, because most of what this SDK does needs no weights: the tokenizer, the
sequence builder, the collator, detection, the presets, the router's DECISION
and the email cleaner are all exercised by two files that are kilobytes, which
is why the cheap parity cell runs in seconds. It also records that
`typed-decisions` is routable here -- Router knows it, its four aliases resolve
-- but has NO recorded end-to-end fixtures, so the decision is tested and the
forward pass against that checkpoint is not. Every claim in it was checked
against the tree rather than copied from the .NET equivalent: the flags, the
paths, the gitignore, the exporter's arguments, the error text when a graph
directory holds neither layout.
CHANGELOG.md in Keep a Changelog form, and the Known limitations section is
the point of it. It names what is not implemented, the 88-in-200,000 softmax
divergence, the 218-in-1.7M float repr spelling, the seven tokenizer
behaviours no shipped checkpoint declares, and -- under its own heading -- the
seven places Java is not Python, each of which was a defect before it was a
lesson.
THE SAMPLES ARE SINGLE-FILE SOURCE PROGRAMS, NOT A GRADLE MODULE, and that is
a decision rather than laziness. A `samples` subproject would inherit the root
build's maven-publish and get a MavenPublication like every other subproject,
and `publishAllPublicationsToCentralRepository` would push it -- which is
exactly why the empty `laya-java-client` module was taken out of
settings.gradle.kts. A version burned on Central cannot be replaced, and
demonstration code has no business being publishable. They still compile in
CI, with -Xlint:all -Werror, so a sample that stops matching the API fails the
build rather than sitting in the tree misleading a reader.
All three were RUN, not just compiled, and running them found two things a
compile could not:
* Routing.java routed an English state, which would have asked a reader to
export a SECOND 1.3 GB graph to see the same thing. It uses a Portuguese
state now and needs one.
* Benchmark.java warmed up 20 times on a batch of 16 -- about 340 forward
passes, 70 seconds on one line, which reads as a hang. The model-backed
figures are 1 warmup and 2 rounds on a batch of 8 now, and the whole run is
34 seconds. A sample that looks hung is a sample nobody finishes running.
What they print, on this machine: detection 0.085 ms on a 42-character state
and 0.496 ms on 3.8 KB; the email cleaner 0.119 ms; one forward pass 1.32 s and
a batch of 8 at 0.93 s per state, on a 322M model on CPU. Benchmark reports the
COLD first call separately rather than averaging it in, because for this SDK
that figure includes the one-time walk of Unicode -- 66 ms -- and quoting it as
throughput is how a port comes to look slower than it is. It also explains why
the per-state batch figure is only modestly better: batching amortises
tokenization, collation and the session call, and none of those is the cost
here.
Quickstart answers `billing` for both an English and a Hindi state using the
same question objects, which is the point of a question set being plain data,
and reads the answers by pattern-matching the sealed `Answer` interface rather
than by looking up a map -- a choice is not a score, and the compiler can check
that every shape is handled.
Routing shows the three parts that fail in different directions: the decision
and its reason string for four scripts, which needs no model at all; the
load-and-evict lifecycle with `maxLoaded(1)` so the eviction is visible; and a
shortlist of 400 labels down to 5 with a deliberately bad embedder, because the
quality of a shortlist is the quality of the caller's encoder and a bad one
removes the right answer before the model can be asked about it.
768 tests pass identically on JDK 17 and JDK 24, all ten README java blocks
still compile, and actionlint with shellcheck is clean on the workflow.
… 21 it does not
The JDK matrix added two commits ago failed on its first real run, in the cell
it was added to cover. `UnicodeTablesTest` asserted that
`Character.isLetter` DISAGREES with CPython over all 1,114,112 code points --
with a comment saying that if a future JDK ever caught up, the test should fail
so the table could be reconsidered rather than kept out of habit.
A JDK has caught up. JDK 21 ships Unicode 15.0 and this port records CPython
15.0.0, so the two letter sets are identical there and the digests match. JDK 17
(Unicode 13.0) and JDK 24 (16.0) both still disagree, which is why only the 21
cell went red.
Catching up is not a reason to remove the table; it is the reason the table
exists. The agreement is two version numbers lining up by accident and it
un-happens on the next release in either direction, so a port that read the
predicate from the JDK would have been right on 21 and wrong on 17 and 24 --
which is precisely the defect this port shipped in its pre-tokenizer, where the
same jar gave different token ids on different JDKs.
So the test is split along the line that actually matters:
* the two properties the JDK has NO equivalent for are still asserted, because
they disagree on every version rather than on old ones. `str.isspace()`
takes U+00A0, U+0085, U+2007 and U+202F and `Character.isWhitespace`
rejects all four -- a different predicate, not a stale one -- and the
canonical combining CLASS is not exposed by the JDK at all, with Mn/Mc (what
the .NET port uses) differing on 1,528 code points.
* whether `isLetter` and `isUpperCase` agree is now RECORDED and printed, not
required. What is asserted instead is version-independent: that the tables
themselves still match CPython, and that not every JDK predicate has caught
up at once -- which would genuinely be the moment to reconsider.
Verified against a real JDK 21 rather than reasoned about: Temurin 21.0.12.1
reproduces the failure (`isLetter agrees with CPython = true`,
`isUpperCase = true`) and passes with the fix.
JDK 17 IS NOW IN THE MATRIX TOO. The build job already tests on 17, but without
`-PtestJavaVersion`, so `TestJvmVersionTest` is disabled there and "17 is
tested" was true but unasserted. The three cells are also three different
Unicode versions -- 13.0, 15.0, 16.0 -- which is the axis this port's whole
character-handling strategy is about, and a two-version matrix would not have
found this.
One thing the failure hid: the old assertion was the FIRST in its test, so when
it failed on 21 nothing else in that cell was ever checked. The model-free suite
now runs clean on all three JDKs, 626 tests each, identically. 769 with the
checkpoints and the fused graph attached.
…can move The drift gate failed on CI and reproduced exactly here, with identical numpy, torch, transformers and tokenizers pins -- so not a toolchain difference. The cosines in `shortlist.json` were rounded to twelve decimals, and twelve is fine until a value lands within the accumulation error of a rounding boundary. Measured rather than assumed: the worst difference between numpy's `dot` and a sequential sum over 4,000 random unit-vector pairs is 2.22e-16, about one ULP near 1, and these vectors are only eight-dimensional. With 194 floats in the file the chance that at least one straddles a twelve-decimal boundary is about 8.3%. At nine decimals it is 0.0085%. The gate was a coin flip rather than a contract, and the comment on `_round_score` said "a precision BLAS blocking cannot move" -- which was simply false and is corrected rather than softened. Nine is three orders of magnitude tighter than any porting error this repository has seen -- a wrong normalisation, a missing clamp or an inverted tie order moves a score in the third decimal, not the tenth -- and seven orders looser than the measured spread. What is still compared EXACTLY is the label order, which is the only thing a consumer observes and the thing a ranking bug moves. `scorePrecisionIsHonest` did its job: it pins the fixture's precision and the comparison tolerance to each other so neither can drift alone, and it failed the moment the fixture changed. Both move together here, along with four comments that asserted twelve decimals in prose -- including the two on the zero-norm and clamp tests, which explain why those are asserted EXACTLY precisely because the rounded comparison cannot see 1.4e-200 or 1.0000000000000002. AND THE GATE NOW SHOWS WHAT DRIFTED. `--check` printed "shortlist.json is stale" and nothing else: not which value, not by how much, not whether it was one float or the whole file. Diagnosing a cross-platform float difference from that took three CI round trips, two of them spent on the wrong hypothesis. On failure the step now regenerates and prints the diff, which costs nothing on the passing path. 626 model-free tests pass on JDK 17, 21 and 24 identically, the three samples still compile with -Werror, and every fixture family matches.
The previous commit was the wrong fix. It lowered the recorded precision from
twelve decimals to nine, which is a real improvement for the `cases` section --
but the values that were actually drifting are in `predicts`, and they had no
rounding whatsoever.
`predict_shortlist`'s result was handed through verbatim:
"shortlist": result["shortlist"],
so the reference's `Ranking` objects kept their raw float64 scores -- sixteen
and seventeen significant digits -- while the same numbers in `cases` went
through `_round_score`. The diff the build lane now prints says it outright:
- 0.5259653204777385 + 0.5259653204777384
- 0.45371261735814744 + 0.4537126173581474
- 0.12186924644473365 + 0.12186924644473371
Sixteen decimals, not nine. Changing the rounding precision could never have
fixed those, because nothing was rounding them. Three values differed in their
last bit between x64 and arm64, which is exactly what a raw float64 cosine does.
THE DEFECT WAS A MISSED CALL SITE, NOT A PRECISION, so the guard is on the
OUTPUT rather than on the code. `_assert_scores_are_rounded` walks the whole
payload and refuses to emit a shortlist fixture holding any float under a
"scores" key that is finer than the recorded precision, at any depth. Reverting
just the one-line fix makes it name all eleven offending values and their paths
instead of writing the file. A grep for `_round_score` found two call sites and
both were correct; what was wrong was a third place that needed one and had
none, and only a check on what was written could see that.
Also swept the other six byte-gated families for the same shape. Two hold
full-precision floats and both are safe: `lang_detect`'s 68 are hand-written
rounding-boundary literals like 0.49999999999999994 that the round-to-int cases
exist to test, and `shortlist`'s remaining 26 are `embed_probe`, which comes
from an FNV-1a hash and integer division -- bit-exact on any platform.
`python_json`'s two are deliberate: that family tests float repr. Nothing else
computes a float through BLAS.
The nine decimals stay. They were not the bug here, but twelve was still a
coin flip for the `cases` section -- 8.3% chance that one of 194 floats straddles
a boundary -- and the measurement behind that is recorded on the constant.
626 model-free tests pass on JDK 17, 21 and 24 identically, every fixture family
matches, and regenerating twice gives identical bytes.
`decode(int[])` and `decode(int[], boolean skipSpecialTokens)`, covering the
ByteLevel, Replace, ByteFallback, Fuse, Strip and Sequence decoders. Decoding
has no Python reference to port -- it lives in the Rust `tokenizers` crate --
so `fixtures/decode_text.json` is recorded from the crate and is the only
specification this port has.
Byte-exact against the reference over both shipped checkpoints: 5,312 ids,
5,312 whole texts, 201,279 window slices and 10,624 special-token sequences,
zero differences.
The SLICES are why the fixture has the shape it does. `predictLong` cuts a
tokenized state into windows at arbitrary offsets and decodes each one, so a
boundary lands inside a multi-byte character routinely. Every whole-text
decode agreed while 629 of 201,279 slices did not: ByteFallback does not
decode lossily, it emits one U+FFFD per BYTE of a run it cannot use, and lossy
decoding emits one per maximal subpart. A fixture of whole texts would have
shipped that.
Four more divergences lived in the generic machinery, where no shipped
checkpoint can reach them -- english declares ByteLevel and multilingual
declares Sequence[Replace, ByteFallback, Fuse], so between them they use four
of the six decoders. Each was confirmed against the crate before it was
changed:
* an absent `decoder` section joins the tokens with a SPACE, not with
nothing. That was wrong for every multi-token decode, in the one branch
written to be permissive rather than to refuse -- so it is where a
third-party checkpoint lands.
* the `<0xNN>` parse is Rust's `u8::from_str_radix`, which is
case-INSENSITIVE and accepts a leading `+`. This rejected both, with a
comment asserting the crate wrote them uppercase only. The comment
described the crate's actual behaviour as a hazard to avoid.
* a Sequence nested in a Sequence hands its chained LIST to the enclosing
sequence's next step. Re-joining it collapsed the boundaries the file is
written around.
* a failed byte run is one piece PER BYTE, so a later per-piece step sees
the boundaries. Invisible on the shipped checkpoints because Fuse is the
next step and re-joins everything.
Decoding is LOSSY on the multilingual checkpoint and that is the contract:
Metaspace prepends its marker, so `decode(encode("Hello world"))` is
`" Hello world"`. Correcting it would re-tokenize every predictLong window
differently from the reference.
Where the crate PANICS -- Strip underflows its own slice indices, which
crosses the FFI boundary as a Rust panic -- there is no reference answer, so
the fixture records which inputs panic and the test asserts this port stays
total instead.
Also here, because the recorded corpus could not see any of it:
* five synthetic reference-recorded tokenizers covering Strip, the absent
decoder, nested Sequences, byte-run piece boundaries and every `<0xNN>`
spelling. Strip had no coverage at all -- replacing its body with a
constant passed all 191 cases -- and neither did any of the seven refusal
paths, which were all removable at once with the suite still green.
* `unverifiable` now looks for a skip marker at any depth. It looked one
level down, and this family groups its sections under `by`, so on a
machine without the checkpoints `--check` reported false drift and a plain
run replaced 75 KB of parity expectations with 7 KB of skip markers.
* the decode family and its tests were in no CI lane. Wired into the parity
tokenizer cell, whose floor goes to 374 -- tighter than the old margin,
which let the smallest class in the cell vanish whole. The model-free
floors go to 625 against 631: 585 and 440 did not bind the new tests.
* the `min-tests` comment claimed the floor protected the window slices. A
count floor cannot see them; the in-suite `>= 400` assertion does.
…in its distribution
`laya-java/gradle/wrapper/gradle-wrapper.jar` arrived with the laya-java CI lane
and nothing checked it. It is a binary that executes at build time, which makes
it the one file in this module a reviewer cannot read, and the workflows run it
five times across five jobs.
The jar in the tree is the genuine Gradle 8.10.2 wrapper -- sha256
2db75c40782f5e8ba1fc278a5574bab070adccb2d21ca5a6e5ed840888448046, computed
locally -- so nothing is wrong today. Nothing would have caught a later swap.
`gradle/actions/wrapper-validation` now runs immediately after checkout in
EVERY job that runs ./gradlew, not one of them: each job checks the tree out
again and runs the jar again, so validating a single job leaves the others
executing an unverified binary. That is three jobs in java.yml (build,
jdk-matrix, parity) and two in release-java.yml (build, central).
`release-java.yml`'s `central` job is the reason this is worth doing rather
than worth noting. It executes the wrapper while holding the PGP signing key
and the Maven Central credentials, and publishes a signed artifact. A swapped
wrapper jar there is not a broken build, it is a signed release built by
something other than Gradle, and a version burned on Central cannot be
replaced.
The wrapper also downloaded its distribution with no checksum, so it trusted
whatever services.gradle.org returned over the wire. `distributionSha256Sum`
now pins gradle-8.10.2-bin.zip to
31c55713e40233a8303827ceb42ca48a47267a0ad4bab9177123121e71524c26 -- the value
Gradle publishes at distributions/gradle-8.10.2-bin.zip.sha256, fetched and
compared rather than copied from a changelog.
The action is pinned to 0b6dd653ba04f4f93bf581ec31e66cbd7dcb644d, which is what
refs/tags/v4 in gradle/actions points at.
Swept the rest of the class in these two workflows while here, and found
nothing further: every `${{ }}` that reaches a shell is `runner.arch` or
`matrix.*`, so no attacker-controlled value (`github.event.*`, `github.head_ref`,
`github.actor`) is interpolated into a `run:` block; no `pull_request_target`;
`permissions: contents: read` on both; artifact uploads are build reports and
fixtures, with no credential paths; every action pinned by SHA.
actionlint clean on both workflows.
`stateRoom`, `budget` and `batchCap`, ported from `laya.common` -- the three
functions `predictLong` needs before it can exist, and the only ones in that
path with no dependency on hooks.
The window size is a correctness question, not a tuning one. A long state is
read in overlapping token windows, each decoded back to text and scored as an
ordinary state, so a window wider than the room the questions leave is
re-truncated on the way in: the tail of every window reaches no model while the
reported span says it did. Once the stride exceeds the real room, consecutive
windows stop touching and the tokens between them are read by nothing at all --
worse than not windowing. So the window is capped at `stateRoom`, and at the
SMALLEST room when several questions are asked, because the windows are one
list of states scored for every question in shared forward passes.
`fixtures/window_plan.json` records it from Python: the `state_room` table per
option count, every branch of `window_budget` including its three warnings and
two refusals, and an 810-point grid for `window_batch_cap`.
The WARNINGS are compared verbatim, not counted. Every clamp here is invisible
to the caller -- their window shrinks, their stride shrinks with it, and the
forward-pass count can triple -- so the warning text is the only thing that
explains a scan that got slower. A port that clamped to exactly the right
numbers and said nothing would pass a test that checked only the numbers. Java
has no ambient warning channel a test can read, so `Budget` carries them as
data as well as logging them.
Two things the case list had to learn. The first version used literal option
counts: it covered every branch on english and reached ZERO of the three
warning branches on multilingual, because `state_room` is a property of the
TOKENIZER -- 88 four-word options leave 150 state tokens on english and 614 on
multilingual. The cases now name the branch they want ("roomy", "mild",
"hard") and the count is resolved per checkpoint by measurement, with a
coverage assertion in the generator that refuses to record a case list which
has stopped reaching the warnings or the refusals. That assertion is what
caught it.
The second: the recorded `state_room` table had a zero-option row. That is
expressible as an internal dict in Python (`crit: {}`) but not through a typed
API that validates its arguments, and it pins nothing one option does not.
Both refusals are IllegalArgumentException and say entirely different things --
"your label set is too big for any window" versus "your stride leaves tokens no
window reads" -- so the test asserts the MESSAGE, not just the type.
16 mutants, 15 killed. The survivor deletes the `window >= configBudget` early
return in `batchCap`; swept over 1,548,519 points where that condition holds,
it makes zero difference, because the arithmetic below reaches the same answer
on its own. It is an equivalent shortcut rather than an untested branch, kept
for the intent it states and documented at the code so the next reader neither
removes it believing it untested nor tries to write the test that cannot exist.
Wired into the parity tokenizer cell in the same change rather than left gated
nowhere, which is how the decode family first landed. That cell's floor goes
374 -> 440 against a measured 444, and the floor was checked to BIND: dropping
the smallest class in the cell leaves 439 and exits 1.
Full suite 1023 tests, 0 failures. Model-free 644 on JDK 17, 21 and 24.
`release-java.yml`'s "The suite was not vacuous" step is a `run: >` folded
block, and the comment explaining the floor was written inside it. YAML folds a
`>` block to a single line and `#` is not a comment inside a scalar, so the
comment reached the shell and silenced the rest of the folded line. Parsed from
the branch, the shell received:
python3 scripts/check_test_results.py laya-java/build/test-results/test
Both arguments were gone. `--min-tests 625` never arrived, so the floor fell to
the script's default of 1 instead of rising to 625; `--allow-aborted` never
arrived either, which on a release runner with no checkpoint makes the gate fail
on the parity aborts instead. One direction is silent and one is loud, and the
step exists to stop a gate that does not bind.
The comment now sits above the `run:` key. The parsed command carries both
arguments again, checked by parsing the workflow rather than by reading it.
`scripts/check_workflow_folding.py` makes it a build failure rather than
something a reader has to notice, and `java.yml` runs it before anything
expensive. actionlint does not report this -- the YAML is valid and the command
is a legal one -- and a `|` block keeps its newlines so the same comment is a
real shell comment there, which leaves the broken and the correct form one
character apart in a diff. The check is standard library only, so it needs no
install step.
Verified three ways: clean on this tree (10 workflows), exit 1 with the defect
re-injected into a copy, and no false positive on a literal block carrying the
same comment.
Scanned every other `run: >` block in the tree for the same shape, as asked.
This was the only one.
…truthiness
An adversarial review of the structured-`decide` commit reproduced eleven
findings. One is a correctness defect, two are divergences from the reference
the commit claimed not to have, one is missing coverage a mutant walked
through, and the rest are documentation and build hygiene. All eleven are
addressed here; nothing else is touched, and the recorded fixture contracts
move not at all -- `fixtures/structured.json` is +277/-0.
**A non-finite `score` projected outside the schema's declared range.** With
no `probabilities` the level came from `(long) Math.rint(score)`, and `(long)`
on a non-finite double is a silent narrowing that `-Werror -Xlint:all` cannot
see: NaN narrows to 0 and +Infinity to Long.MAX_VALUE. For a field declared
`{"type": "integer", "minimum": 1, "maximum": 3}` that projected 1 for a NaN
and -9223372036854775808 for an infinity -- no exception, no gate flag, and
nothing in the `Decision` saying so. CPython refuses both (`ValueError` for a
NaN, `OverflowError` for an infinity). It is refused here too, in `Plan.value`,
which is the one arithmetic path `expect` did not cover.
**`NaN`/`Infinity` were spelled `json.dumps`'s way on the `%r` and `str`
path.** `PythonJson.repr(double)` reproduces the `dumps` spelling, which is
right for `dumps` and wrong for a refusal message and wrong for a choice
LABEL -- the label is what the model is shown and what the answer comes back
keyed by, so `{"nan": ...}` on one side and `{"NaN": ...}` on the other is a
different answer key for a cross-language caller. `Decisions.pythonRepr` now
gives `nan`/`inf`/`-inf`; `PythonJson.repr` keeps the `dumps` spelling it
serves, and now says so. The root cause was a false comment claiming `repr`
gives `"NaN"`. It does not; `repr(float('nan'))` is `nan`.
**`anyOf` reproduced one case of Python's truthiness, not the rule.** The
reference spells it `prop.get("anyOf") or prop.get("oneOf")`; the port spelled
it "not a list, or an empty one". The two agree on an empty list and disagree
on everything else: a truthy non-array was sent through to `oneOf` -- refusing
`{"anyOf": "abc"}` as "unsupported schema" where the reference says "got 0",
and mapping `{"anyOf": "abc", "oneOf": [...]}` to a question the reference
refuses outright. `truthy` is now Python's, over all seven falsy JSON shapes.
The one case left divergent is an `anyOf` that is truthy and not iterable, a
number or `true`, where the reference raises `TypeError` from inside its own
comprehension; that is recorded as the single exception to the
message-for-message rule rather than left implied.
**Two of fifteen independent mutants survived.** An empty `anyOf` falling
through to `oneOf`, and an empty `description` meaning "generate one", were
both correct and both asserted by nothing. Two fixture cases now kill them,
recorded from the reference like every other expectation.
Also: an ungated four-argument `decide`/`decideBatch`, because the
five-argument form is ambiguous on a bare `null` and "no gate" could only be
written `(Double) null` -- which the suite itself did six times, and no longer
does; the `forwarded` datum, recorded for all seven call cases and asserted in
none, is now the pin for the `min_confidence` forwarding divergence, structural
half included; a null schema is refused with the reference's own message, the
one case of its top-level type check that a typed signature does not already
cover; and the three real behaviour divergences that lived only in private
member comments, which javadoc strips, join the published class-level list,
along with `decide_batch`'s per-state routing, which this port does not have.
`LAYA_CHECKPOINTS` and its two siblings are declared as task inputs.
`environment(...)` is not one, so toggling a variable left `test` UP-TO-DATE
and handed back the OTHER lane's results -- a measurement of the wrong thing
that looks exactly like a measurement of the right one. Verified both ways: the
toggle now re-runs, and an unchanged re-run is still UP-TO-DATE.
Measured on the final tree. JDK 17 without checkpoints: 798 registered, 14
abort, 784 run, 0 failed. With both checkpoints: 1,198 registered, 3 abort,
1,195 run. `-PtestJavaVersion=24`: 798 registered, 13 abort, 785 run. The
model-free floor moves 771 -> 779 at both sites, keeping the 5-test margin;
DecisionsTest is 99 of the 784, so with it out of the tree the suite is still
green at 685 run and the floor exits 1. The historical `--min-tests 625`
comment is untouched. Eight mutants -- the two survivors re-expressed against
the new code, plus one per fix -- all compile and all die, each on the test
written for it. The reviewer's 112-case mapping differential goes from 102
identical / 10 differing to 104 / 8, and all eight remaining are now claimed
divergences in the published list; the 41-case projection differential has no
silently-wrong value left. The CHANGELOG no longer compares a registered count
against a run count.
The previous commit said the config check "runs before any probability is
compared" in four places, and that claim was the stated reason a wrong
checkpoint/graph pairing was safe to document rather than prevent. It was false.
`TypedDecisionsParityTest` carried no `@TestMethodOrder`, JUnit 5 applies no
ordering of its own, and the class shares one `Agent` through a static field, so
whichever test ran first opened the weights. Measured with the english
checkpoint symlinked in as `typed-decisions/`: the identity check ran NINTH of
thirteen, after all eight probability comparisons, and 11 of the 13 tests passed
under that pairing -- not the "seven of the eight" the disclosure claimed, since
all three batch configurations passed too and the batch factory therefore
contributed nothing to the defence.
The check is now asserted inside `agent()`, on the shared agent as it is opened,
so it gates every consumer instead of racing them; the failure is cached so a
mispairing costs one 1.7 GB open rather than fourteen. `@Order` puts the named
test first so the report reads in the order the gating happens. Measured on the
same pairing now: 4 tests register, 3 fail, both factories fail to register at
all, and the cell's exact `min-tests` floor fails on the count as a second
signal.
The clamp and bucket assertions had zero power over that pairing and the comment
above them implied otherwise. The two checkpoints ship `temperature_by_options`
IDENTICALLY -- every key, every digit, `choice:11+` at 0.10058280825614929
included -- so the keyset assertion, the per-bucket loop, the raw-value
assertion and the `TEMP_MIN` clamp all pass unchanged under the wrong weights.
They catch a port bug, which is worth having, and nothing else. Only `max_len`
(1024 against 512), `head_max_len` (256 against 192) and the fitted
`temperature` triple discriminate -- and the triple never reached a forward
pass, because every recorded question resolved to a bucket both files ship. So
`unbucketed-score` is recorded: a six-level score question, which `temp_bucket`
maps to `score:6-10`, a bucket neither checkpoint ships. The lookup misses and
`temperature[QTYPE_SCORE]` is what scales those logits, 1.0374 here against
1.2514 on english. Pinned: mutating the per-type fallback to 1.0 is killed by
`unbucketed-score` alone and survives the previous commit's suite untouched,
13 tests 0 failures. Under the mispairing with the identity gate lifted, two
cases now discriminate on their numbers where one did before --
`payment_priority.answer_confidence` 0.3645 against 0.3362, nineteen times
`over-budget`'s 0.3046 against 0.3061.
`model` was recorded on every batch result and compared only on the single-state
path. It is compared on both now; mutating one recorded batch `model` is killed
by `batch_size=all sort=false` and survives the previous suite.
The model-free count in the CHANGELOG read 626. Measured with every `LAYA_*`
variable unset: 703 register, 686 run, 17 abort. CI's own floor had already
drifted 54 above the documented figure, which is independent evidence it was
stale, and this commit moved the two bullets below it. Both floors are
re-derived exactly: 686 for the `build` lane and 687 for the JDK matrix, which
passes `-PtestJavaVersion` and so runs `TestJvmVersionTest` instead of skipping
it. Each binds with no slack -- 687 and 688 respectively fail on the real
results.
`LAYA_TYPED_ONNX_GRAPH` is in the build's deliberate pass-through list now. It
worked without being there only because the Test JVM inherits the ambient
environment, which is the quiet inheritance that list exists to replace, and it
is the one of the four variables that two of the three parity cells set to `''`.
The cache-key justification for splitting the parity cell was wrong.
`prepare_checkpoint.py` pins a SINGLE `HF_REVISION` for every checkpoint and is
itself inside the key's `hashFiles(...)`, so a bump invalidates every cell at
once; "each checkpoint's graph is invalidated by its own revision bump" names
something that does not exist. What stops a restored pre-bump cache from being
reused is the `.laya-revision` stamp `stamped()` reads. The split stands on the
other two reasons, and the per-checkpoint key is still worth having for restore
granularity.
The golden records that stamp now, so a `HF_REVISION` bump cannot re-record this
family in silence: `graph` was only a basename, the same string at every
revision. `gen_fixtures.py --check` is the comparator -- pinned by bumping the
stamp beside the graph, which takes `--check` from exit 0 to exit 1.
`predict.json` is deliberately left without one: adding the field would make the
committed file disagree with the generator, and re-recording it needs the
multilingual graph.
The headline `{refund: 0.9077, replace: 0.0923}` was cited in five places and
appears nowhere in the committed fixture, so no reader could re-derive it. It is
replaced by `off-workflow`'s `{billing: 0.8967, other: 0.1033}`, which is in the
file and which the mispairing reproduces.
"Graph present, checkpoint absent" was the one generator mode that raised a bare
`FileNotFoundError` out of `laya/onnx_agent.py` rather than writing the
actionable `skipped` marker every sibling builder writes. It writes the marker.
Verified on the final state. `build -x test` passes with -Werror and javadoc.
Model-free: 703/686/17 plain and 703/687/16 under `-PtestJavaVersion`, 0
failures, on Corretto 17 and Corretto 24. Typed cell: 14 tests, 0 failures, 0
skipped on both, identity first. `gen_fixtures.py --check` exits 0 in every mode
and the fixture diff is purely additive -- `revision` plus the one new case, 56
lines, no deletion, config and batch sections byte for byte.
Twelve defects, every one of them reproduced before it was touched, every one pinned by a test that a reverting mutant kills. TWO JAVADOC CLAIMS WERE FALSE, and both were load-bearing. `HookRegistry.Scope.close` documented its backward scan as unobservable "because the copies are the same object". They are, so the multiset left behind is the same either way -- but the ORDER is not, and order is the one thing about a hook list this port calls a contract. With `addHook(own)`, `addHook(other)` and then `hooksInstalled(own)` the list is `[own, other, own]`: dropping the last match leaves `[own, other]` and dropping the first leaves `[other, own]`, so a forward scan silently reorders two hooks the caller installed itself and changes which one sees the other's work. `Hooks.dispatch` said a deadline "protects the request and not the process". It protects neither. What it bounds is THE WAIT. The thread keeps running and keeps holding the live, mutable `PredictContext` of a call that has moved on without it, and `PredictContext`'s fields were plain, so there was not even a happens-before edge once `join` expired. Measured over 60 calls whose start hook overran a 50 ms deadline by 30 ms and then assigned results: 60 of 60 late writes were accepted, and `ctx.usage` ended up describing a different answer from `ctx.results` in 60 of 60 -- because `around` had already totalled the usage of the answer it saw. In the runs where the write landed in the few microseconds before `around` read the results back, the CALLER got the abandoned hook's answer. The reference has the identical hazard, so the wording is now right about what the deadline does. It is also now narrower than the reference, deliberately: a thread is MARKED when its deadline expires and every `PredictContext` mutator refuses a write from a marked thread, so the same 60 calls disagree 0 times. That is a narrowing and not a proof, and the javadoc says so with the number -- with the overrun cut to 1-5 ms, so that the write and the deadline collide, 2 to 17 of 60 late writes still landed. The context's mutable fields are `volatile` for the other half of the same problem. FOUR CONTRACTS WERE DOCUMENTED AND UNTESTED, each confirmed by a mutant that survived the whole suite before this commit: that an `Error` is never swallowed whatever `raiseErrors` says (widening the catch to `RuntimeException | Error` survived); that `withoutDefaultHooks` nests (`set(false)` in place of `set(previous)` survived, and the single existing scope test could not tell); that the abandoned thread is a daemon (`setDaemon(false)` survived); and that the interrupt flag is re-asserted before the wait's exception is wrapped (deleting the `interrupt()` survived). The deadline itself was pinned by nothing at all -- multiplying every deadline by twenty and adding half a second survived -- so it now has a behavioural bound, loose by 15x so a loaded machine cannot turn it red. THE GREP-PARITY CLAIM WAS WRONG TWICE. `call` promises that "a caller grepping their logs across the two runtimes should find the same line", and both of the two lines dispatch reports differed. A deadline went through `%s` on a double, so a one-second deadline read `exceeded 1.0s` where CPython's `%g` gives `exceeded 1s` -- and a whole number of seconds is the common case for a deadline. `Hooks.seconds` is C's `%g` written out, because Java's own `%g` is not C's; `fixtures/hooks.json` now records CPython's rendering per recorded deadline so the comparison is generated rather than typed, and the formatter was checked by hand over 23 values besides, byte-identical on all 23. And `getSimpleName()` is the EMPTY STRING for an anonymous class, so `Hooks.onPredictStart` and `onPredictEnd` -- the two advertised low-friction hooks, and the shape a telemetry hook takes -- reported `laya: hook .on_predict_start failed`, naming nothing, in the one line `raiseErrors(false)` leaves behind. They are now the reference's own named classes, underscore included, because the name is data; a caller's own anonymous hook falls back to the binary name's last segment, a case the reference cannot reach. TWO GUARDS THAT LET THE SUITE TEST NOTHING. JUnit does not fail an empty `@TestFactory`, so a `hooks.json` section that emptied produced a factory with no tests in it. The count floor was the only guard and it was not enough: with the smallest section emptied, the `build` lane ran 793 against a floor of 794 and caught it by exactly one test, while `jdk-matrix`, which runs one test more, landed exactly ON 794 and exited 0. `HooksTest.section` now refuses an empty or missing section and names it, and the two lanes carry floors one apart so the arithmetic is not what saves them. Separately, the timeouts factory routed any unrecorded value into the branch that asserts only about the reference and never calls `validateTimeout`, so appending a representable value to the reference's `HOOK_TIMEOUT_VALUES` silently stopped testing Java; it now fails on a value it has been told nothing about, the way its sibling factory already did. NULL HANDLING AND TWO NUMBERS. "a hooks entry must not be null" was unreachable from every single-hook entry point, because each wrapped the argument in `List.of(...)` first and the caller got a bare `NullPointerException` from inside the JDK. `Hooks.Totals` summed per-state `int` usage into an `int` and wrapped silently -- three results of a billion input tokens each totalled -1294967296, which no Python int can produce and which `usage` is what a metering hook bills from; it counts in `long`. And `predictBatch(List.of(), null)` returned an empty list before the hooks were wired in and NPEs now, which was an unmentioned change: `PredictContext` refuses a null `states` or `questions` by name instead, which is what the reference does for an empty batch as much as for any other. Process-wide default hooks were correct in-tree and protected only by two classes remembering to clear them in teardown. That is now structural: an auto-registered `DefaultHooksIsolation` clears them after every test in the module, pinned by a class that deliberately does not remember. Mutation: 38 mutants, 38 killed, each asserted present in the file and compiling before the suite ran. Eighteen of them are the previous commit's own claims re-run against this tree, so that fixing these twelve could not resurrect one. The model-free floor goes from 794 to 813 on the build lane and 814 on jdk-matrix (831 registered, 817 and 818 run), proven to bind by shrinking the suite: without `AgentHooksTest` 809 run and the gate exits 1, without `HooksTest` and its stubs 695 run and it exits 1.
… branch
Brings the three remaining `laya-java` items onto `feat/laya-java-routing`,
each already carrying its own adversarial review and the fixes that review
demanded:
* `typed-decisions` end to end, plus the identity check that now gates the
forward pass instead of running ninth of thirteen.
* `Decisions.decide` / `decideBatch` against a JSON schema, plus the refusal
of a non-finite score that used to narrow silently into the declared range.
* the prediction hooks, plus the twelve defects an independent sweep found in
them -- six surviving mutants, three documented contracts with no test, and
a deadline that bounded the wait rather than the request.
Three things could only be settled here, on the merge, because no branch could
see them alone.
`build.gradle.kts` keeps BOTH halves of its environment block. One review found
`LAYA_TYPED_ONNX_GRAPH` missing from the pass-through list; a different review
found that `environment(...)` is not a Gradle task input, so toggling a lane
left `test` UP-TO-DATE and handed back the previous lane's results. Neither fix
supersedes the other and taking one side would have dropped a real one.
`scripts/gen_fixtures.py` is merged structurally, not textually. Git aligned the
hunks on `}` and `try:` and cut them mid-function, which does not parse. The two
sides add 27 and 17 top-level blocks with no name collision, and both edit only
`FAMILIES`; the result is their union, 18 families, asserted to parse with no
duplicate definitions.
The test-count floor is re-derived rather than reconciled. Each branch raised it
against its own base -- 779, 686/687, 813/814 -- and every one of those leaves
over a hundred tests of slack against the merged 917, so each would have been
decorative. Measured here: 934 register and 917 run with no checkpoints, 918 in
the `jdk-matrix` lane because `-PtestJavaVersion` is what enables
`TestJvmVersionTest`, and 1,328 of 1,334 with both checkpoints present. The
floors are EXACT, with no margin, which replaces a margin of 4: three classes in
this lane run exactly one test, and no positive margin can bind a one-test
class. Proved by deleting `WindowPlanTest` -- the suite stays green at 916 run,
the exact floor exits 1, and 813, 779 and 686 all exit 0.
`README.md` and the CHANGELOG state what is true after the merge rather than
what each branch knew: hooks, `decide`, `predictLong` and `ConfidenceGate` all
exist, so only the HTTP client module and Android remain unimplemented.
`java/unreleased-lock`, high, in the blocking `security-extended` suite, on `Hooks.dispatch` -- the one place where a stuck lock wedges every hooked call rather than one. The code was not the naive form. `lock()` and `unlock()` were already paired across a `try`/`finally`. The defect is that `policy.lock()` was CALLED TWICE, once for each half. `Policy` is a record, so both calls return the same monitor today and the path is not reachable; nothing in the type requires that, and locking one instance while unlocking another leaves the first held with no owner able to release it. Read once into a local now. Neither semgrep nor an independent 14-mutant sweep over this file saw it. It is not a property of any line -- it is a property of which paths reach the unlock, which is the whole reason CodeQL is in the lane. This is the second unreleased lock it has found in this module that the pattern scanner called clean. Measured: 14 findings before, 12 after; `unreleased-lock` goes to 0. The suite is unchanged at 934 registered, 917 run, 0 failed. `Hooks.seconds` also takes an explicit guard for a missing exponent marker. It is unreachable from a `Duration`, whose `toNanos()` is a long so the value is always finite -- but the method is public so a caller's own failure sink can emit the same line, and `%.5e` of a non-finite double renders "Infinity", which holds no 'e', so `substring(0)` would hand the whole word to `Integer.parseInt`. Stated plainly: this does NOT clear the `uncaught-number-format-exception` finding, which merely moves to the new line number -- CodeQL cannot prove the substring is numeric whatever guards it. It is now the same accepted class as the two pre-existing ones in `PythonJson`. `%g` parity with CPython re-verified in both runtimes afterwards, byte-identical on ten values including 1e-07. The four `java/unused-parameter` hits on `Hook`'s default no-op methods are left alone: an unused `ctx` is what a no-op default IS, and `Router.java:111` is the same finding already accepted at base.
Comment-only; `git diff` shows zero non-comment lines changed. Both `check_test_results` steps already carried the right floors, 917 and 918, measured on the merged tree. The prose above them did not. The build lane's block still read "703 register here, 686 run ... so it executes 687" -- the typed-decisions branch's figures, sitting four lines above a floor of 917. The jdk-matrix lane's block still read "814, not the build lane's 813 ... 818 run against a shared floor of 813" -- the hooks branch's figures, above a floor of 918. Each was correct on the branch it was written on and wrong the moment four branches merged, which is the same failure this PR documents in the code it reviews: a justification that reads as measured while the number it justifies has moved underneath it. Mine, this time. Both now state the merged measurement, and the jdk-matrix one says why the two lanes differ by exactly one rather than repeating a margin argument that no longer applies. Also records what CI has since settled. That comment said the exact floor was prepared without a JDK 21 to test on, so whether 21 registers the same count as 17 and 24 was an argument -- no JDK-version-gated assumptions exist in the suite, and 17 and 24 agreed -- rather than a measurement. All three lanes have now run green at these floors, so it is a measurement, and the comment no longer claims otherwise. Kept as deliberate history, not stale: the "(779, 686, 813)" list of what the four branches each set against their own base, and the 794/793 account of the shared floor that let an emptied fixture section pass invisibly.
…nload pressure
`.NET / parity (multilingual)` has failed twice on this branch while passing on a branch
that adds no parity cell, so it is not the flake I first called it, and the mechanism
implicates this change.
From the failing job's own export step:
Warning: You are sending unauthenticated requests to the HF Hub.
Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Fetching 5 files: 100%|##########| 5/5 [00:04<00:00, 1.03it/s]
Five files including a 34 MB `tokenizer.json`, in four seconds, unauthenticated. `hf-xet`
returned a TRUNCATED file and did not raise: the download step reported success and the
failure surfaced ninety seconds later as `EOF while parsing a value at line 1 column 0` from
inside a tokenizer constructor -- "line 1 column 0" being an empty file. 19 of 302 tests
failed, every one a multilingual tokenizer case, while english, router and typed-decisions
all passed in the same run. A code defect does not empty one checkpoint's file and leave the
others intact.
This matrix gained a third weight-downloading cell in this change, and `dotnet.yml`'s parity
matrix runs four of its own against the same repo in the same workflow run, with
`max-parallel` unset on both. Peak was seven concurrent unauthenticated fetches of one repo.
`max-parallel: 1` here drops that by two and costs wall-clock on a lane that is already
advisory.
Stated plainly: this is a MITIGATION of pressure this change added, not a fix for the
defect. The defect is that nothing verifies a download before using it, so a truncated fetch
becomes a confusing parse error three minutes downstream instead of an immediate, obvious
failure. The fixes for that are an `HF_TOKEN` for the higher rate limit, or a size or digest
check at the fetch -- both in `laya-dotnet/tools/` and `dotnet.yml`, which this PR does not
touch, so neither is bundled here rather than widening it into another module's lane.
Earlier I attributed this to a transient download and said this PR could not be the cause.
The twice-on-this-branch, pass-elsewhere pattern does not support that, and the comment in
the workflow now records what actually happens instead.
`.NET / parity (multilingual)` failed twice on this branch and passed on a sibling. From the failing job's own log: the hub rate-limited an UNAUTHENTICATED request and `hf-xet` wrote a 0-byte 34 MB `tokenizer.json` without raising. `snapshot_download` reported success; the failure surfaced ninety seconds later inside a .NET tokenizer constructor as `EOF while parsing a value at line 1 column 0`, a message that names no download. 19 of 302 tests failed, every one a multilingual tokenizer case, while english, router and typed-decisions passed in the same run. Three parts. `verify_snapshot(directory, label, required)` at the three download sites CI reaches -- `laya-dotnet/tools/regen_golden.py`, `laya-dotnet/tools/export_onnx.py` and `laya-java/scripts/prepare_checkpoint.py` -- refusing a snapshot whose requested files never arrived, or that holds a 0-byte file, unparseable JSON, or a `.safetensors` too small for a header. `required` is the load-bearing half: walking what is present cannot see a file that never came, and an empty directory passes a walk. `max-parallel: 1` on both parity matrices. Every cell needing weights fetches the same repo, and four in `dotnet.yml` plus three here in one run is what provoked the throttle. Peak concurrent fetches drops 7 to 2. Costs 7m38 of serial wall-clock here and 19m12 there, on lanes that are advisory. `HF_TOKEN` on the three fetching steps -- scoped to those steps, not workflow-level, because a workflow `env` reaches `pip`, `gradle`, `semgrep` and every third-party action in both files. The review of the first version of this found two defects that would have made it worse than the bug, and both are worth recording. `verify_snapshot(..., name)` was written where the enclosing parameter is `name_or_path`, so every Hugging Face download would have raised `NameError` and all four .NET parity cells would have failed unconditionally -- and because this change alters the files the export caches hash, the cache miss guaranteed the path ran. `ruff --select=F82` finds it in under a second, so `ci.yml` now runs that group over the three SDK script directories as well as `laya/`. Split into a second invocation: those directories carry pre-existing F401/F811 import debt that is not worth churning here, while F82 is a bug class rather than style. Worse, the guard REJECTED a healthy checkpoint. With `local_dir=`, `huggingface_hub` keeps bookkeeping under `<dir>/.cache/huggingface/download/` whose `.lock` files are legitimately 0 bytes; english has no subfolder, so the verified directory was the `local_dir` itself and `jvm parity (tokenizer)` would have failed every run. `.cache/` is now skipped, and the real english checkpoint is verified to pass. A fail-closed check that refuses correct input is the one failure mode it must not have. Also from review: an unset `HF_TOKEN` secret expands to `""`, and the explicit `token=` call site passed it through, producing `authorization: Bearer ` and `httpx.LocalProtocolError: Illegal header value`. That is a client-side refusal, not a 401, so no server probe finds it, and every fork PR would hit it. Normalised to None at the call site. The `laya-ts` copy of the guard is dropped: `regen_golden.py` invokes that exporter with `--model-dir`, which short-circuits its download, so it is unreachable from CI and the file ships to npm. Verified: the guard accepts the real english checkpoint including its 0-byte `.lock` files, and refuses a 0-byte `tokenizer.json`, a truncated one, a missing required file and an empty directory, identically at all three sites. Both ruff invocations exit 0 on this tree and exit 1 on the reinstated NameError. `check_workflow_folding` clean, actionlint clean, all three workflows parse. Model-free suite unchanged at 934 registered, 917 run, 0 failed. Stated plainly, since the guard is not a complete one: a `.safetensors` truncated above 1024 bytes still passes, and the real one is ~2.3 GB. Closing that needs the hub's `siblings` sizes or a `safetensors` header length check, which is a larger change than this failure warrants.
The previous commit said a `.safetensors` truncated above 1024 bytes "still passes" and that
closing it needed the hub's `siblings` sizes or a header-length check, "a larger change than
this failure warrants". The first half is true of `verify_snapshot`; the second half was not
measured, and it is wrong.
`safetensors` already validates that the tensor byte ranges cover the file. Measured: the
842 MB english `model.safetensors` truncated to 500 KB raises
SafetensorError: Error while deserializing header: incomplete metadata,
file not fully covered
in the exporter, seconds after the fetch, naming the actual defect. Nothing needs closing and
no follow-up is owed.
That is the whole distinction this guard exists for. The 0-byte `tokenizer.json` was NOT
self-diagnosing -- `EOF while parsing a value at line 1 column 0`, from a .NET constructor, in
a test, ninety seconds later, naming neither a file nor a download. A loader that already says
"file not fully covered" needs no help; one that says "EOF at line 1 column 0" does.
So the 1024-byte test stays a floor for a file too short to hold the 8-byte length prefix and
a header at all -- where the loader's own error would be about a malformed header rather than a
truncated download -- and the comment now says that instead of leaving a reader to infer a gap.
Comment-only; zero non-comment lines changed.
…arrying NandhaKishorM#938 was merged at 561c496 -- the `Tokenizer.decode` commit -- so everything after it is still open: `hooks/`, `Decisions.java` and `fixtures/typed_decisions.json` are absent upstream. upstream/main has since moved 158 commits, which is what made this conflict. Of the 51 commits on this branch, 27 have a same-subject twin upstream: they landed with that merge under different SHAs. 24 are genuinely new. 13 files conflicted. Resolved per file rather than by preference. For ten of them the test is mechanical -- upstream has not touched the file since my work merged, so this side is upstream's content plus the new work, and `--ours` loses nothing. Three needed a decision: `decode/Decoder.java` takes UPSTREAM. My own comment in that file deferred the question: an empty action block padded to two and published {0.5, 0.5}, and the comment said changing it "belongs in its own change against that test rather than buried in this one; raised with the maintainer separately". The maintainer decided -- it throws now -- and taking this side would have reverted their decision on a question I handed them. `fixtures/lang_detect.json` takes UPSTREAM. It was re-recorded by 7a2f1bb after the Python lang fix landed as NandhaKishorM#1013, so upstream's copy pins the fixed behaviour and this branch's copy pins the bug. `.github/workflows/java.yml` takes OURS, verified rather than assumed. Upstream added one `gradle/actions/wrapper-validation` step in dc72133; this side has it in all four jobs that run `./gradlew`, each before that job's first `./gradlew`. Parsed the YAML to confirm, because dropping a security step is the worst thing a merge can do quietly. Two tests then failed, and BOTH fail on pristine upstream/main at e08843b with nothing from this branch in the tree. They are inherited, not caused here. `DecoderTest` is fixed here, because its assertion had stopped matching the code it guards. `Decoder.actionProbabilities` throws on a zero-width block while the test asserted {0.5f, 0.5f}. The implementation is the half that is right: 0.5 is a number no model produced, published into the field callers gate escalation on, and the reference raises there too -- `laya/onnx_agent.py` indexes `act[offset + r, 0]`, so zero columns cannot return. The test now asserts the refusal and its message, and a mutant that disables the refusal fails it. `LanguageDetectionParityTest` is NOT fixed in this commit, and the reason is worth stating. `english-note-with-acronym-line-and-foreign-line` expects `pt` and the port answers `fr`: the fixture was re-recorded from the fixed `laya/lang.py`, but `lang/LanguageDetection.java` was never updated to match, so the two have diverged in exactly the way the parity gate exists to prevent -- and in exactly the way NandhaKishorM#1013's own PR body warned they would if the fixture moved without the port. Porting that fix is a real change against 116 recorded cases and belongs in its own commit, not inside a merge. Verified: `gradle build -x test` clean under -Werror with javadoc; 924 tests complete; the only failure left is the LanguageDetectionParityTest case above.
…pected The Python side merged the caps bars (NandhaKishorM#1013 and its follow-up); the Java port never got them, so LanguageDetectionParityTest has been failing on main for the one recorded case where an all-caps line of English abbreviations outvoted a genuinely Portuguese line. MON, LA, EST, COM and DES are hockey teams, states, time zones and radio bands that collide with French stopwords. Before this, the bare-acronym line was named fr and won the whole-state verdict. Ported from laya/lang.py: - SHOUTED_MIN_STOPWORD and SHOUTED_MIN_WORD, with the reference's sweep recorded on each. - shouted(): a cased segment written entirely in capitals. The uppercase half is not redundant -- a caseless script has no lowercase either, and would otherwise be held to a bar its tokens cannot clear. Both halves go through the recorded case tables, not the JDK's isUpperCase/isLowerCase, which disagree with Python on 240 code points. - shoutedEvidence(): a word of SHOUTED_MIN_WORD letters, and then either non-English diacritics or a matched stopword of SHOUTED_MIN_STOPWORD letters. - nonLatinCarries(), named rather than left inline, because the caps bars ask the same question to decide what to keep out. - the bar applied in namedProseLanguage, and again in analyseText so the whole-state verdict obeys the module's own rule that an acronym is not a word. Vetoing there leaves the text undecided, so looksNonEnglish still decides it. - only acronym-shaped runs are blanked, via upperLetterRuns(). Blanking every all-caps run deletes emphasis capitals, which are ordinary in prose and usually sit on the words that carry the sentence. blankUpperRuns() itself is unchanged: it is a recorded helper the fixture pins, so analyseText filters before calling it. - leafNonEnglish(): a veto is not a verdict of English. A field whose named language the prose scan disbelieves now falls through to the diacritic branch instead of being dropped. The fixture is not regenerated. lang_detect.json was re-recorded upstream from the fixed reference and is the target here. ShoutedAcronymBarTest asserts the behaviour directly, because the fixture cannot: of the seven mutants used to check it, only one is caught by the parity suite at all. Deleting the analyseText block, blanking every all-caps run, reverting the leaf fall-through, dropping the non-Latin exemption and loosening either bar by one letter all leave LanguageDetectionParityTest green. Suite: 934 tests 1 failed -> 947 tests 0 failed. Parity 116/116.
…n produce
`gen_fixtures.py --check` went stale on `confidence_gate.json` after the merge, and following
that to its cause found a behaviour divergence rather than a recorded-message drift.
Upstream `cb85656` changed `laya/confidence.py` from "a key must be a string" to "a key must
name a bucket an answer can produce". This port only ever checked for null, so it accepted keys
the reference refuses:
'choice:3-5' 'default' 'choice:99' 'nonsense'
reference accept accept REFUSE REFUSE
this port accept accept accept accept
now accept accept REFUSE REFUSE
That is worse than a wrong message. A threshold filed under `choice:99` is never applied to
anything, because no answer resolves to that bucket -- the caller believes they gated a decision
and nothing is gated. Accepting the key the reference refuses turns a loud error into exactly
that silence, which is what this gate exists to prevent.
The allowed set is derived from the same spelling `optionBucket` emits -- three question types
crossed with the four size bands, plus `default` -- rather than written out, so the two cannot
drift. The message is matched character for character including `%r` quoting and `None` for a
null key, verified against the merged Python on the same four inputs.
`fixtures/confidence_gate.json` is regenerated on **CPython 3.12**, which `java.yml` pins. One
leaf moved, the refusal message. 3.11 would have rewritten `unicode_version` to 14.0.0 and every
Unicode digest with it, which the maintainer lost a commit to; the fixture records 15.0.0 and
3.12.2's `unicodedata` is 15.0.0. All 18 families now match.
The fixture could not have caught this. Its recorded refusal uses the key `1`, an int, and this
port's signature is `Map<String, ? extends Number>`, so that case is unrepresentable here and the
comparison skips it. A bad STRING key is representable and nothing recorded reaches that path, so
`ConfidenceGateTest` sat at 26/26 green over the gap. `ConfidenceGateBucketKeyTest` covers it,
including a case that feeds every bucket `optionBucket` can emit back in as a key so the allowed
set and the bucketing function cannot diverge. A mutant that drops the check to the old null-only
test kills it: 3 tests, 1 failure, compiled clean.
Floors re-derived, which the maintainer asked for explicitly: 950 register, 933 run, so 933 and
934. Proven to bind by deleting a one-test class -- 929 run, the floor exits 1 -- and every piece
of prose above each floor moved with the number, because a justification above a changed figure
is the defect this branch has already shipped twice.
…atters `max-parallel: 1` on both parity matrices cost more than it bought. Measured from this PR's own cells, serialising takes the parity critical path from 10:56 to 30:08 -- java 2:24 + 5:14 + 8:05 against 8:05 parallel, dotnet 7:52 + 7:28 + 10:56 + 3:52 against 10:56 -- which is nineteen minutes added to every run to reduce the chance of a fetch being throttled. Two keeps the peak well below the seven concurrent unauthenticated fetches of one repo that provoked the throttle, at roughly half the wall-clock. The reason this is safe to relax is that serialising was never the fix. `regen_golden.py`, `laya-dotnet/tools/export_onnx.py` and `laya-java/scripts/prepare_checkpoint.py` now refuse an incomplete snapshot AT THE FETCH -- a file that never arrived, a 0-byte one, unparseable JSON -- so a throttled fetch fails immediately with the file named, instead of surfacing ninety seconds later as `EOF while parsing a value at line 1 column 0` from inside a tokenizer constructor. `max-parallel` only changes how often that path is taken; the guard is what makes taking it survivable. The earlier commit said as much and then chose the expensive half anyway. Comment-only apart from the two values: `git diff` shows four non-comment lines, the old and new setting in each workflow. Note for whoever watches the next run: this PR also changes all three files that are hashed into the export cache keys, so the first run after it is cold in every parity cell regardless of this setting. That cost is one-time and is not what this commit addresses.
A choice's options are positional: option N renders Nth and logit N is read back
as the Nth label. So the criteria map's iteration order decides both what is
asked and how the answer is labelled -- and `choice` accepted any `Map`, copying
it into a `LinkedHashMap` that faithfully preserved whatever order the caller's
map happened to have.
Measured on JDK 17, options written `refund, escalate, ignore`:
LinkedHashMap refund, escalate, ignore
HashMap ignore, escalate, refund
Map.of varies per JVM run
`HashMap` is exactly reversed, so `probabilities().get("refund")` reported the
logit that belonged to `ignore`. That much is at least reproducible, because
`String.hashCode` is specified. `Map.of` is worse: it salts its table per JVM, so
three runs of one program gave `[ignore, escalate, refund]`,
`[escalate, ignore, refund]` and `[refund, escalate, ignore]` -- the same question
answering differently after a restart, with nothing in the output to say so.
`Map.copyOf` returns that same implementation, so copying a `LinkedHashMap`
through it destroys the order it was built to carry.
`choice` now refuses such a map and names the fix. A denylist rather than an
allowlist, because a false refusal breaks working code: Guava's `ImmutableMap` and
`Collections.unmodifiableMap(aLinkedHashMap)` both iterate in a defined order this
check cannot see. Checked by `getClass()` rather than `instanceof`, because
`LinkedHashMap` IS a `HashMap` and an `instanceof` test would reject the one map
the method most wants to accept. A single-option map is always accepted: one
option cannot be out of order.
`Question.choiceOf(instructions, Map.entry(label, description), ...)` is the
counterpart for callers who would otherwise reach for `Map.of`: no intermediate
collection, so there is no order to lose. It refuses a duplicate label, which a
map would have resolved by keeping only the last -- asking a question with fewer
options than were written. The check is `containsKey` rather than the return of
`put`, because a legitimate criterion may be null and then `put` returns null for
a duplicate too.
Pinned by ten tests, and both halves are pinned by a mutant: neutering the guard
fails four of them, removing the duplicate check fails one.
Floors moved with the count, which is exact by design and has no margin:
java.yml model-free 933 -> 943
java.yml jdk-matrix 934 -> 944 (+1 is --require-passed TestJvmVersionTest)
release-java.yml 680 -> 943
The release floor was stale enough to be no gate at all: against a measured 943 a
suite could have lost 263 tests and still passed it. Its comment said "685 of the
699 registered tests run without a checkpoint", which is also now the measured
943 of 960.
build -Xlint:all -Werror clean
model-free suite 960 registered, 943 run, 0 failed
real-weights e2e 1367 tests, 0 failed, 0 errored
check_workflow_folding 10 workflows clean
The e2e ran both local checkpoints and the exported graph: TokenDecoderTest 195,
LanguageDetectionParityTest 116, TokenizerParityTest 86, WindowPlanTest 66,
SequenceParityTest 52, PredictLongTest 49, DecoderParityTest 40,
PredictParityTest 9. `TypedDecisionsParityTest` ran 4 and skipped 3: the
typed-decisions checkpoint is not on this machine.
… the new default
`3cf26cb` ("default to the multilingual checkpoint for undecided text") changed
`laya/router.py` without regenerating `laya-java/fixtures/router.json`. GitHub
tests a PR merged with main, so the stale fixture failed here even though nothing
on this branch caused it:
gen_fixtures: router.json is stale
##[error]Process completed with exit code 1
`router.json` is stale on `upstream/main` by itself, so this failed on every open
PR. Timing: b7cc4da passed every job at 11:16, 3cf26cb landed at 14:14, 253b5c2
failed three jobs at 19:20.
Regenerating it exposed the real divergence the fixture exists to catch: both
ports still defaulted to english.
laya-java/fixtures/router.json regenerated, 24 lines
Router.java Builder.defaultCheckpoint ENGLISH -> MULTILINGUAL
LayaRouter.cs Default English -> Multilingual
Identified English still routes to english on both; only the undecided default
moved, which is all 3cf26cb changed. Two Java tests and two .NET route cases
asserted the old default and are updated. `RouterLifecycleTest` was asserting
"plain english text here now" -> ENGLISH; `analyse` gives that string
language=null, undecided=true, so it was never identified English and only landed
on english through the default.
Also fixes three findings from an adversarial review of 253b5c2:
HIGH -- `orderIsUnspecified` used `getClass()` equality, so a subclass slipped
through: `class Mine extends HashMap {}` was accepted and published option 0's
probability against option 2's label. `instanceof` now, which is safe because the
ordered types return before that point -- and the comment justifying `getClass()`
was simply wrong about why.
HIGH -- `Shortlist.rank` copied the caller's map into a `LinkedHashMap` before
calling `Question.choice`, so the guard could never fire there. Order decides
which labels survive the cut in `Shortlist`, not just how they are numbered, so a
`HashMap` dropped a different option. Validated before the copy now.
HIGH -- the README claimed `choice` "refuses a map whose contract leaves the order
unspecified". It does not: `Collections.unmodifiableMap`/`synchronizedMap`/
`checkedMap` are the same wrapper class whichever map they wrap, so a wrapped
`HashMap` is accepted. Reaching the wrapped map needs `--add-opens` into
`java.base`. The README and javadoc now say what is and is not caught.
Three tests added for the two holes, and five stale measured counts in
java.yml's comments corrected.
build -Xlint:all -Werror + javadoc clean
model-free suite 963 registered, 946 run, 0 failed
gen_fixtures --check 18/18 match
check_workflow_folding 10 workflows clean
Floors: java.yml 946 and 947, release-java.yml 946.
Not verified locally: the .NET change. `dotnet` is not installed on this machine,
so `LayaRouter.cs` and the two route cases are reasoned from the same Python
reference the Java fix was verified against, and CI is the first run of them.
fa4631b to
9cd5de4
Compare
…point
`LayaRouter.Default` moved to multilingual with the reference, but the committed
routing goldens still recorded english on the default path: 8 entries reading
`using default (english)`. CI regenerates these from Python on every run and the
drift check against the committed copy is only informational, so the fresh
goldens said multilingual while the committed ones said english -- which is the
mechanism behind the two failing .NET jobs.
Regenerated with `tools/regen_golden.py --checkpoint routing`: 7 `model` and 7
`reason` fields, nothing else. One `using default (english)` remains, for the case
that sets `default="english"` explicitly. A second generator run is a no-op, so
the output is stable.
Verified in a container, since this machine has no dotnet
(mcr.microsoft.com/dotnet/sdk:10.0, arm64, SDK 10.0.401):
dotnet build Laya.slnx -c Release 0 Warning(s), 0 Error(s)
Laya.Tests, CI's five exclusions Total: 709, Failed: 0, Errors: 0, Skipped: 0
Note for anyone reproducing it: `dotnet test --solution Laya.slnx` reports "No
test projects were found" in that image -- `IsTestProject` evaluates empty. The
project is xunit.v3 with `OutputType=Exe`, so the test assembly is its own runner;
running `bin/Release/net10.0/Laya.Tests` with `-class-` per excluded class is the
equivalent and works.
`Router` dispatched nothing. `Hook` already declared `onRoute`, `onLoad` and `onEvict`, `Hooks.Event` already had the three constants and `Hooks.dispatch` already existed -- nothing ever called them, so the one thing a router-level hook could observe that an agent cannot was unobservable. Now dispatched: onRoute after a checkpoint is chosen onLoad after a build, not on a cache hit onEvict from maxLoaded pressure, unload and unloadAll predict one start/end pair for the whole route-and-answer call `route` wraps a new private `decide`, so the event fires once from a single place. `decide` returns from six branches -- explicit model, explicit task, a typed-decisions workflow match, a language hint, detection, the configured default -- and a hook that fired from five of them would be worse than none. Dispatched OUTSIDE the router lock. The build publishes and `evictLocked` runs while `lock` is held, and a hook is caller code: one that blocks would hold the lock with it and park every other thread asking for any checkpoint. An `Outcome` holder records what an acquire did and the caller dispatches once it is outside. `evictLocked` already returned the evicted names; the return value was being discarded. `RouterHooksTest.aReentrantHookDoesNotDeadlock` covers it. `Router.predict` reuses `Hooks.around` rather than writing out a start/end pair, so `ctx.skip`, the error-before-end order, a failing hook being attached to the real failure instead of replacing it, and `elapsed_ms`/`usage` all behave exactly as they do on an `Agent`. The lease is taken inside the envelope, so this call's `on_load`/`on_evict` land between its start and end events. Not honoured, and said so in the javadoc and the README: `ctx.maxLen` / `ctx.headMaxLen` from a router-level start hook. `Agent.predict` takes no budget arguments, so there is nowhere to put them. An agent-level hook can still set them. `AsyncHook.of(hook, executor)` runs a hook on a chosen thread -- a framework request scope, an actor, a UI loop -- and WAITS for it, as the reference does (`laya/hooks.run_coroutine_sync` drives the coroutine to completion). Waiting is what makes it safe: `PredictContext` is mutable and per-call, so a late callback would read another call's state, and a thrown exception would be lost instead of reaching the hook policy. An optional `Duration` fails a callback that overruns; a non-positive one is refused rather than read as "never wait". NOT included: hooks on `predictLong`. It was implemented and reverted, because `AgentHooksTest.predictLong runs no hook` caught it. A scan is sized before any hook can run, so a start hook replacing `ctx.states` desynchronises `results` from `starts`, which `aggregate` walks in parallel -- answers would be attributed to the wrong window offsets. That test's message names the precondition: the reference's start probe and budget check have to be ported first. The README now gives that reason instead of listing predictLong as an unexplained gap. build -Xlint:all -Werror + javadoc clean model-free suite 973 registered, 956 run, 0 failed real-weights e2e 1380 tests, 0 failed, 0 errored check_workflow_folding 10 workflows clean 13 new tests, and three mutants kill them: never dispatching ROUTE fails one, never dispatching EVICT fails one, and an AsyncHook that does not wait fails three. Floors moved with the count, which is exact by design: java.yml 943 -> 956 and 944 -> 957, release-java.yml 943 -> 956.
… hooks
Two of the commit's own mutants survived its tests, so two tests were not testing
anything; the AsyncHook javadoc claimed the opposite of what the code did; and
`lease` could strand a lease it had taken.
AsyncHook: an overrunning callback is abandoned, not merely cancelled.
`CompletableFuture.cancel` never interrupts a task already running on an executor,
so the callback kept going and its writes to a finished call's `PredictContext`
landed. `PredictContext.abandon(Thread)` is the guard this repo already built for
that and `Hooks` already uses it for a hook that overruns `hooks_timeout`; it was
simply unused here. The task now reports the thread it runs on and that thread is
abandoned on overrun, so its late writes are refused. The javadoc's safety
paragraph was inverted on the one path where it mattered and now says what
actually happens -- including that the callback keeps the executor's thread until
it returns.
Router.lease: a throwing `on_load` stranded the lease. `acquire` increments
`slot.leases`, and the `Lease` that would release it was constructed after the
dispatch, so a hook failure left a retired-but-never-closed session with nothing
able to release it. `build` guards the same shape with `release(built); throw`, and
`load` releases before dispatching; `lease` was the omission.
Router.preload: built checkpoints through the legacy no-Outcome overload, so it
dispatched nothing at all -- against the README's own table.
A throwing `Agent.close()` ate both events. `closeAll` runs before the dispatch
and rethrows the first failure, so a build that published and then failed to close
the evicted agent lost a real `on_load` and a real `on_evict`. The dispatch is in
a `finally` now, at `load`, `unload` and `unloadAll`.
Two tests that could not fail:
* `unloadAll`'s `on_evict` loop was unpinned, and `close()` goes through it -- the
commonest eviction in any real program. Deleting the loop kept the suite green.
* `aReentrantHookDoesNotDeadlock` could not fail for its stated reason. `lock` is a
`ReentrantLock` and the callback ran on the dispatching thread, so it re-entered
whether or not the lock was held; wrapping the dispatch in `lock.lock()` kept the
suite green. Replaced with a probe that reaches the router from a SECOND thread,
which is the only thing that can tell the difference. Both mutants now fail.
Documentation that was wrong or missing:
* `maxLoaded()`'s javadoc had been orphaned by inserting `hooks()` between the
comment and its method. `Xdoclint:none` is why the javadoc task stayed green.
* `Hook`'s own javadoc still argued AsyncHook was deliberately not ported -- the
first javadoc a hook author reads, contradicting the shipped class.
* A process-wide default hook sees TWO predict pairs per `Router.predict`, the
router's and the agent's, so a metric summing `usage()` double-counts. A
throwing `on_evict` fails `close()` and drops the remaining evictions. Calling
back into the router during `close()` throws. `concurrent(false)` with an
`AsyncHook` deadlocks: serial dispatch holds a lock across a callback that is on
another thread. `on_route` carries its own context, so `runId` differs from the
predict pair's. All now in the README.
build -Xlint:all -Werror + javadoc clean
model-free suite 980 registered, 963 run, 0 failed
real-weights e2e 1387 tests, 0 failed, 0 errored
check_workflow_folding 10 workflows clean
Floors re-measured: java.yml 963 and 964, release-java.yml 963.
Not fixed, and worth a follow-up: a hook calling `router.predict` from
`on_predict_start` recurses until the stack runs out, and `load()` can hand back an
agent a hook unloaded while the event was being dispatched.
9cd5de4 to
c9d7800
Compare
|
Merged. This is the rebase I asked for after 0.3.29, and it arrives with the hooks half of the SDK plus three things I did not ask for and am glad to have. The dependency scan is the one I want to call out. I read the The truncated-download fix is the one that has been costing this repo real time. The empty 34 MB multilingual On the size, and this is the same note as last time rather than a new complaint: 38,000 lines arriving after a rebase is a lot to review as one piece, and what makes it reviewable is that the fixtures are generated and gated, the runtime is asserted against the Python reference, and each commit stands alone. That is a real property and it is why I take these. It is still worth landing the next piece on its own. All 85 suites pass on the merged tree, the 82-row real-weights decision diff is unchanged, and |
35 pull requests from 8 contributors. * LAYA_EXTRA_MODELS registers extra checkpoints on laya-serve (#1047, #919) * fine-tuning prints its optimizer-update budget before a run (#1014, #963) * the Kaggle DDP notebook converges on the shared training primitives (#1017, #887) * evals writes redirected output as utf-8 (#1045) * the server serves again after a lifespan restart (#1049) * Java router lifecycle hooks, plus an OSV dependency scan (#1046) * twenty-four documentation corrections, each verified against the code Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(java): router hooks review fixes that #1046's merge did not include
feat(java): dispatch router lifecycle hooks, and add AsyncHook
The gap
Routerdispatched nothing at all.Hookalready declaredonRoute,onLoadandonEvict,Hooks.Eventalready had the three constants, andHooks.dispatchalready existed — nothing evercalled them. So the one thing a router-level hook can observe that an agent cannot was
unobservable: which checkpoint was chosen, and when one was built or dropped.
What now fires
onRouteonLoadonEvictmaxLoadedpressure,unload, andunloadAllonPredictStart/onPredictEndroutenow wraps a privatedecide, so the event fires once from one place.decidereturns fromsix branches — explicit model, explicit task, a typed-decisions workflow match, a language hint,
detection, the configured default — and a hook that fired from five of them would be worse than
none.
Dispatched outside the router lock
This is the part that decides whether a hooks feature is safe. The build publishes and
evictLockedruns whilelockis held, and a hook is caller code: one that blocks on an HTTP call,a queue put or a lock of its own would hold this router's lock with it, and every other thread
asking for any checkpoint would park behind it.
So an
Outcomeholder records what an acquire did and the caller dispatches once it is outside.evictLockedalready returned the evicted names — the return value was simply being discarded.RouterHooksTest.aReentrantHookDoesNotDeadlockinstalls a hook that calls back into the router andproves it completes.
Router.predictreusesHooks.aroundRather than writing out a start/end pair. That envelope already owns the parts that are easy to get
subtly wrong: a start hook answering through
ctx.skip, the error event firing before the endevent, a failing hook being attached to the real failure instead of replacing it, and
elapsed_ms/usagebeing filled before the end hooks read them. The lease is taken inside it, sothis call's
on_load/on_evictland between its own start and end events.Not honoured, and said so in both the javadoc and the README:
ctx.maxLen/ctx.headMaxLenset by a router-level start hook.
Agent.predicttakes no budget arguments in this SDK, so thereis nowhere to put them. An agent-level hook can still set them.
AsyncHookAsyncHook.of(hook, executor)runs a hook on a chosen thread — a framework request scope, anactor, a UI loop — and waits for it. That matches the reference, where an async hook's coroutine
is driven to completion on a background loop (
laya/hooks.run_coroutine_sync); it is not a way tomake a slow hook free.
Waiting is what makes it safe.
PredictContextis mutable and belongs to one call, so a callbackstill reading it after the call moved on would see another call's state, and a thrown exception
would be lost rather than reaching the hook policy. An optional
Durationfails a callback thatoverruns; a non-positive one is refused rather than read as "never wait".
Deliberately not included: hooks on
predictLongIt was implemented, and then reverted, because the repository's own test caught it:
That test pins the behaviour on purpose and its message names the precondition — "until the
reference's start probe and budget check are ported".
predictLong's javadoc gives the reason: ascan is SIZED before any hook could run. Handing the windows to hooks as
ctx.stateslets a starthook replace them, which desynchronises
resultsfromstarts— andaggregatewalks those two inparallel, so answers would be attributed to the wrong window offsets. The reference needs four
pieces of machinery to let hooks in safely.
The README's Status section now gives that reason instead of listing
predictLongas anunexplained gap.
Verified
13 new tests —
RouterHooksTest7,AsyncHookTest6 — and three mutants kill them:RouterHooksTest1 failureRouterHooksTest1 failureAsyncHookdoes not waitAsyncHookTest3 failuresThe e2e ran both local checkpoints and an exported graph;
AgentHooksTestpasses 8/8 there, whichincludes the
predictLong runs no hooktest, so the revert holds under real weights too.TypedDecisionsParityTestskips 3 of its cases: that checkpoint is not on this machine.Test floors moved with the count, which is exact by design and carries no margin:
java.yml943 → 956 and 944 → 957,release-java.yml943 → 956.Scope and risk
is resolved. The only behavioural change for such a caller is that
routenow goes through onemore private call.
Router.hooks()is additive, andAsyncHookis a new class.Adeshina Lasisi <keemsisi@gmail.com>.The change
.github/workflows/java.yml,.github/workflows/release-java.yml,laya-java/README.md,laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java,laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java,laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java,laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java— 7 files changed, 652 insertions(+), 14 deletions(-).diff