Skip to content

test(java): follow #1315's English words in two stale router tests - #1323

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-java-router-english-assertions
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-java-router-english-assertions

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Two hand-written assertions in laya-java's model-free suite still ask for the answer a plain
English state gave before #1315's English function words were carried into this port. Both now
fail, on main at 0a05a450 (release 0.4.14), in every JDK cell:

file was now
laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterLifecycleTest.java::closeFreesAndSeals Checkpoint.MULTILINGUAL for "plain english text here now" Checkpoint.ENGLISH
laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterTest.java::abstainingHintFallsThrough reason().startsWith("Latin script") the recorded reason "English Latin text", exactly, plus the model and the presence of a detection

laya/, the fixtures and the port's production code are untouched. This is the Java test catching
up with the reference, not a change to it.

Why

93dcdf9d ("SDKs: carry #1315's 82 English function words into all four ports") added the words to
LanguageTables.java, regenerated lang_tables.json / lang_detect.json / router.json, and
updated the reference's own tests (tests/test_router.py gained
check("route/english question words …", Router().route(text).model, "english")). Two assertions
that hard-code the old detection answer were left behind, so the lane has been red since 12:45Z:

Run 39955674443 (push, main 0a05a450, "release: 0.4.14")
  jvm build + model-free tests        failure   1182 completed, 2 failed, 13 skipped
  jvm model-free tests (jdk 17)       failure
  jvm model-free tests (jdk 21)       failure   RouterLifecycleTest.java:775
  jvm model-free tests (jdk 24)       failure   RouterTest.java:521

The workflow itself still reports success because the two jobs that went red, build and
jdk-matrix, set continue-on-error: true -- the file's own header says why: "ADVISORY, not
blocking … When this lane goes red the fix is a follow-up port, not a revert of the Python change."
That follow-up port is this PR. ci.yml, go.yml, rust.yml and dotnet.yml are green on the same
commit, so this is Java-only: the other three ports had their recorded probes updated in 93dcdf9d
and assert nothing about the old answer.

The port answers correctly; the assertions are what is stale — three independent witnesses:

  • Running the suite here with the base assertions prints the actual value, not an error in the code:
    AssertionFailedError: expected: <multilingual> but was: <english>.
  • RouterTest::every recorded route matches, model and reason -- the fixture-driven replay of all 47
    recorded routes rows -- passes at base, including the row lang-agnostic-C-falls-through, same
    state string and same code that names no language, which records
    "model": "english", "reason": "English Latin text".
  • The reference says the same thing at 0a05a450: Router().route("plain english text here now")
    → english, "English Latin text"; analyse(...)["language"] == "en", so the removed comment
    "Undecided, not identified English (analyse gives language=null)" is now a false statement.

The prefix form was worth replacing rather than re-pointing, because "Latin script" is also the
prefix of the reason detection gives when it cannot identify a language (laya/router.py::route:
"Latin script, language not identified and no non-English letters; …"). It proved the hint stayed
out of the decision only while plain English stayed undecided. Exact comparison to the recorded
reason is what the same file already does for its other detection case (routingIsPure asserts
"English Latin text").

How it was verified

Gradle could not run here: the pinned wrapper is 8.10.2, which refuses to start on the only JVM this
Mac has (26.0.1), and the build's JDK 17 toolchain is not provisioned. So the suite was compiled and
launched directly — javac --release 17 (which is what the build pins options.release to) with the
onnxruntime jar the module declares, then the JUnit Platform launcher, with
-Djunit.jupiter.extensions.autodetection.enabled=true and src/test/resources on the classpath
exactly as the build configures the Test task. Reproduced in
laya-bench/java_model_free.sh <worktree> (outside the repo).

# base assertions, pristine 0a05a450 worktree
$ laya-bench/java_model_free.sh …/laya-answers-base
  1182 tests found / 1181 started / 13 aborted / 1166 successful / 2 failed
    RouterTest.abstainingHintFallsThrough   expected: <true> but was: <false>   (RouterTest.java:521)
    RouterLifecycleTest.closeFreesAndSeals  expected: <multilingual> but was: <english>  (:775)
  FAILED_COUNT=2

# this branch
$ laya-bench/java_model_free.sh …/laya-java-router-english
  1182 tests found / 1181 started / 13 aborted / 1168 successful / 0 failed
  FAILED_COUNT=0

1166 + 2 → 1168 + 0: the two failures clear and nothing else moves. found, started and aborted
are unchanged at 1182 / 1181 / 13, so no test was added, renamed or lost — which matters because the
lane asserts check_test_results.py --min-tests 1168 and this run meets that floor exactly, the same
figure the workflow's own comment records.

The four new or changed assertions were each mutated to check they carry weight; every arm failed:

arm mutation result
m1 RouterTest: ENGLISH → MULTILINGUAL on the new model assertion 1 failed
m2 RouterTest: reason literal "English Latin text" → "Latin script" 1 failed
m3 RouterTest: assertTrue(fallback.detection() != null flipped 1 failed
m4 RouterLifecycleTest: ENGLISH → MULTILINGUAL 1 failed

Compile hygiene, since build.gradle.kts applies -Xlint:all -Werror to test compilation too:

$ javac -Xlint:all --release 17 -cp <onnxruntime + junit jars> -d out $(find src/main/java src/test/java -name '*.java')
compile rc=0
warnings naming RouterTest.java or RouterLifecycleTest.java: 0

(the run does print dangling-doc-comments warnings in four untouched src/main files — that is a
lint category JDK 26 added, not one JDK 17's -Xlint:all has, so it is an artifact of the local
compiler rather than something CI sees.)

Fixture contract, which is the gate a Python-side change has to pass and this is not: no laya/ file
changed, and gen_fixtures.py --check on this branch is byte-identical to the same command on a
pristine 0a05a450 worktree (stdout empty, rc=1 both, stderr identical after normalising the tree
path). The two families it calls stale locally — lang_detect.json, python_json.json — are this
machine's environment noise, recorded before this change and reported by the pristine tree too; CI's
own run of that step on main prints 20 family/families match laya (9 unverified).

Out of scope, deliberately

  • ShoutedAcronymBarTest's and defaultCheckpointDecidesUndecided's claims about undecided text
    were re-checked against the reference rather than assumed: "Quero cancelar" still analyses to
    language=None, so those tests still say what they claim. Only the two above were stale.
  • No change to laya/lang.py, LanguageTables.java or any fixture: the wordlist change is the
    reference's, this branch only follows it.
  • The advisory continue-on-error policy in java.yml is a maintainer decision, not something this
    PR should reopen.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

…er assertions main is red on

`93dcdf9d` carried the 82 English function words into `LanguageTables.java` and
regenerated the fixtures, but left two hand-written assertions asking for the
answer a plain English state used to get:

- `RouterLifecycleTest.closeFreesAndSeals` expected MULTILINGUAL for
  "plain english text here now". The port answers english -- measured, not
  inferred: running the suite with the base assertion prints
  `expected: <multilingual> but was: <english>`.
- `RouterTest.abstainingHintFallsThrough` accepted any reason starting with
  "Latin script", which is also the prefix of the reason detection gives when it
  cannot identify a language. Once the state is identified the assertion is not
  the test's point any more, so it now compares detection's recorded reason
  exactly -- the same pair the `lang-agnostic-C-falls-through` row of
  router.json records -- and pins the model and the presence of a detection too.

`laya/` and the fixtures are untouched: this is the Java test catching up with the
reference, not a change to it.
thread3d added a commit to thread3d/laya that referenced this pull request Oct 24, 2026
`AgentConfig` kept the `double[]` and the two `Map`s its parser passed in, and
`temperatureByOptions()`/`langTemperatures()` handed the internal maps straight
back out. A caller that built the config from its own collections — or that kept
a config it was given — could retune a loaded checkpoint after the fact, and
every later prediction would quietly use the edited temperature. `temperature()`
already cloned, so this was an inconsistency rather than a policy.

Copied at the boundary instead: `clone()` for the array, `Map.copyOf` for the
maps, and a compact constructor on the `LangTemperatures` record because its
`double[]` is a mutable component. Both parsers already pass immutable maps and
`Map.copyOf` returns such a map unchanged, so the load path pays nothing new.

Verified: ./gradlew test (JDK 17, -Xlint:all -Werror) 1182 tests, only the two
inherited RouterTest/RouterLifecycleTest failures of NandhaKishorM#1323; jcheck's
EscapingMutableInternal (J0603) drops from 16 to 11 with all five AgentConfig
hits gone.
thread3d added a commit to thread3d/laya that referenced this pull request Oct 24, 2026
`open()` never closed its `OrtSession.SessionOptions`: they are an input to
`createSession`, not something a live session keeps, so every load leaked one
native options structure for the life of the process. That is invisible until a
caller reloads checkpoints and native memory keeps climbing.

While there: in the split form (`encoder.onnx` + `head.onnx`), a failure creating
the head session closed nothing. `close()` is only reachable through a
`LayaSession`, and that path never constructs one, so the encoder outlived the
failed load. It is closed now, with its own failure attached as suppressed, the
way `close()` itself reports a second failure.

`OrtEnvironment` is deliberately untouched: it is ONNX Runtime's process-wide
singleton, and closing it would break every other session in the process.

Verified: ./gradlew test (JDK 17) 1182 tests, only the two inherited failures of
NandhaKishorM#1323; the sample programs still compile and jcheck reports 0 diagnostics for
them against the rebuilt classes.
thread3d added a commit to thread3d/laya that referenced this pull request Oct 24, 2026
… and releasing it

In `predict`, the `languageFor(...)` call sat between the lease acquisition and
the `try (Lease held = lease)` that releases it; in `predictBatch`, the
`contexts`/`indexOf` allocations did. Neither throws today, but an
`OutOfMemoryError` from either would leave the lease held with no owner able to
release it — the same shape as the unreleased lock CodeQL found in `Router.build`.

`languageFor` now runs inside the try; the two collections are built before the
acquisition. The statement after taking a lease is the one that releases it.

Verified: ./gradlew test (JDK 17) 1182 tests, only the two inherited failures of
NandhaKishorM#1323 (which is the language-detection expectations, not this path).
@NandhaKishorM
NandhaKishorM merged commit c7eb30a into NandhaKishorM:main Oct 24, 2026
50 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged as c7eb30a. Thank you, this was the only red lane on main and your diagnosis matches what the logs show exactly.

I had reached the same two assertions independently before I found this PR, and yours is the better change, so I dropped mine. Two reasons. You also assert fallback.model(), which my version left out. More importantly, your comment makes the point I had missed: "Latin script" is also the prefix detection reports when it cannot identify a language, so the old startsWith check only demonstrated that the abstaining hint stayed out of the way for as long as plain English happened to stay undecided. Comparing the reason exactly, and pinning the model and the presence of a detection alongside it, is what the test was there to say. That is a tightening, not just a retarget.

Confirming the chain independently, for the record:

  • Pre-fix(lang): count question words and auxiliaries as English evidence #1315 laya/lang.py on these two states returns language=None with language_undecided=True. Current laya.lang.analyse returns language='en', is_english=True, language_undecided=False. So Router moves off the undecided branch and onto its last one, ENGLISH with reason "English Latin text".
  • laya-java/fixtures/router.json already records that exact pair for the same state under lang-agnostic-C-falls-through, which is why the four jvm parity jobs stayed green the whole time while these two hand-written assertions did not. Good catch citing that row.
  • "plain english text here now" and "plain english words here now" are the only two places in the tree carrying the old expectation. The other hits on those phrases are explicit model= or task= cases that never reach detection, and the other three ports assert nothing about the old answer, so this really is Java only.

All four previously red jobs are green on your branch, and the lane is rebuilding on main now.

Two things this exposed that are on me, not on you:

  1. build and jdk-matrix are continue-on-error: true, so the Java workflow reported success for the whole 0.4.14 release while 2 of 1182 tests were failing. The header explains the intent and I am not going to flip it unilaterally, but advisory should not mean invisible, and a release that ships with a red job inside a green lane is the failure mode worth closing. I will look at surfacing the job conclusions separately from the lane's.
  2. 93dcdf9 was my commit. It updated the reference's own tests and regenerated the fixtures, and it should have swept the hand-written port assertions in the same pass. Your PR is that sweep.

NandhaKishorM pushed a commit that referenced this pull request Oct 24, 2026
`AgentConfig` kept the `double[]` and the two `Map`s its parser passed in, and
`temperatureByOptions()`/`langTemperatures()` handed the internal maps straight
back out. A caller that built the config from its own collections — or that kept
a config it was given — could retune a loaded checkpoint after the fact, and
every later prediction would quietly use the edited temperature. `temperature()`
already cloned, so this was an inconsistency rather than a policy.

Copied at the boundary instead: `clone()` for the array, `Map.copyOf` for the
maps, and a compact constructor on the `LangTemperatures` record because its
`double[]` is a mutable component. Both parsers already pass immutable maps and
`Map.copyOf` returns such a map unchanged, so the load path pays nothing new.

Verified: ./gradlew test (JDK 17, -Xlint:all -Werror) 1182 tests, only the two
inherited RouterTest/RouterLifecycleTest failures of #1323; jcheck's
EscapingMutableInternal (J0603) drops from 16 to 11 with all five AgentConfig
hits gone.
NandhaKishorM pushed a commit that referenced this pull request Oct 24, 2026
`open()` never closed its `OrtSession.SessionOptions`: they are an input to
`createSession`, not something a live session keeps, so every load leaked one
native options structure for the life of the process. That is invisible until a
caller reloads checkpoints and native memory keeps climbing.

While there: in the split form (`encoder.onnx` + `head.onnx`), a failure creating
the head session closed nothing. `close()` is only reachable through a
`LayaSession`, and that path never constructs one, so the encoder outlived the
failed load. It is closed now, with its own failure attached as suppressed, the
way `close()` itself reports a second failure.

`OrtEnvironment` is deliberately untouched: it is ONNX Runtime's process-wide
singleton, and closing it would break every other session in the process.

Verified: ./gradlew test (JDK 17) 1182 tests, only the two inherited failures of
#1323; the sample programs still compile and jcheck reports 0 diagnostics for
them against the rebuilt classes.
NandhaKishorM pushed a commit that referenced this pull request Oct 24, 2026
… and releasing it

In `predict`, the `languageFor(...)` call sat between the lease acquisition and
the `try (Lease held = lease)` that releases it; in `predictBatch`, the
`contexts`/`indexOf` allocations did. Neither throws today, but an
`OutOfMemoryError` from either would leave the lease held with no owner able to
release it — the same shape as the unreleased lock CodeQL found in `Router.build`.

`languageFor` now runs inside the try; the two collections are built before the
acquisition. The statement after taking a lease is the one that releases it.

Verified: ./gradlew test (JDK 17) 1182 tests, only the two inherited failures of
#1323 (which is the language-detection expectations, not this path).
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants