Skip to content

fix(lang): describe the state in analyse, and round rates like Python - #1330

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
cyberspace-cs:fix/lang-state-script-fields
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
cyberspace-cs:fix/lang-state-script-fields

Conversation

@cyberspace-cs

Copy link
Copy Markdown
Contributor

What

Two fields of analyse()'s result described something other than the state it was given. Both are fixed in laya-ts/src/lang.ts, with a new laya-ts/tests/state-script-fields.test.ts.

1. A per-value override replaced the state's script fields with the leaf's. When the value pass flips isEnglish, the result now keeps the state's script, scriptProfile, diacriticRate and nonLatinFraction, and overrides only language, isEnglish and languageUndecided — the same three Python's analyse overrides.

2. The reported rates rounded a tie away from zero. round4 now rounds the double's own decimal expansion, so a tie goes to the even digit as Python's round() does, and a value that only looks like a tie after * 10000 is not treated as one.

Why

analyse returns the script fields of the state. They were coming from one leaf instead:

const note = "Please read the attached invoice carefully and let me know if you need anything else. ".repeat(20);
analyse({ note, msg: "Мне нужен счет на оплату." });
// before: { script: "cyrillic", scriptProfile: { cyrillic: 1 }, nonLatinFraction: 1, isEnglish: false }
// after:  { script: "latin", scriptProfile: { latin: 0.9859, cyrillic: 0.0141 }, nonLatinFraction: 0.0141, isEnglish: false }

The state is 98.6% Latin, and scriptProfile(stateText(state)) already said so for the same input, so analyse disagreed with the module's own exported function. Anything reading analyse(state).script or .nonLatinFraction — a "detected language" label, a per-script metric — was told the ticket was entirely Cyrillic, and the isEnglish: false beside it does not reveal that the route came from one field.

The rates: one non-English letter in 32 characters is 0.03125 exactly.

python -c "import laya; print(laya.detect_language('é' + 'a'*31)['diacritic_rate'])"   # 0.0312

Math.round(0.03125 * 10000) / 10000 is 0.0313. The obvious fix — a half-to-even branch on x * 10000 — is wrong in the other direction: 1 in 4000 is 0.0002500000000000000052 as a double, which Python rounds up to 0.0003, while 0.00025 * 10000 lands on 2.5 exactly and a tie rule would report 0.0002. Both directions are pinned by tests.

How it was verified

cd laya-ts
npm install
npx tsc -p tsconfig.json
npx vitest run

44 files / 1224 tests passed. The new file fails 7 of its 9 tests on main and passes all 9 here (git stash push laya-ts/src/lang.ts then re-run).

Differentially against Python's laya.lang.analyse, over 6 616 string states and 1 504 structured ones (nested dicts/lists, 11 languages and scripts, a 4 000-character cap, byte inputs):

corpus disagreements before after
strings 6 1
structured 108 0

Every disagreement in that run was a field in this PR's scope. The one left is MON DES EST LA: Python has an all-caps bar (_shouted_evidence) that the port does not have, so it reports the state as undecided-English where the port names French. That is a routing change rather than a field-description bug, so it is not in this PR — happy to do it separately if you want the port to match there too.

Checklist

  • Focused on one change (both are analyse returning a field that does not describe the state; I can split them into two PRs if you would rather review the rounding on its own)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed — no API change: the types are unchanged and the values now match the reference

`analyse` promises the script fields of the state it was given. When the
per-value pass flipped `isEnglish`, the TS runtime returned that leaf's
detection whole, so a 98.6% Latin ticket carrying one Russian line reported
`script: "cyrillic"` with `scriptProfile: {cyrillic: 1}` and
`nonLatinFraction: 1` -- three fields that contradict
`scriptProfile(stateText(state))` for the same input, and that a caller
reading `analyse(state).script` cannot tell apart from the `isEnglish` flag
beside them. The override now answers `language` and `isEnglish` only, as
Python's `analyse` does.

The reported rates also broke a tie away from zero, where Python's `round()`
takes the even digit: one non-English letter in 32 characters is 0.03125
exactly, reported as 0.0313 here and 0.0312 by the reference. Scaling by
10000 also invents a tie that the double is not -- 1 in 4000 is
0.0002500000000000000052, which Python rounds up and a `* 10000` tie rule
rounds down -- so the digits come from the double's own expansion.

Differential-tested against Python's `laya.lang.analyse` over 6 616 string
states and 1 504 structured ones: the structured disagreements drop from
108 to 0 (the two remaining rows in that run feed raw bytes through a JSON
harness), the string ones from 6 to 1. The one left is the all-caps bar,
which the port does not have; not addressed here.
@NandhaKishorM
NandhaKishorM merged commit 8ff4fb3 into NandhaKishorM:main Oct 24, 2026
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
fix(laya-ts): stop reading all-caps lines as French or Portuguese

Add/add conflict against #1330, which had landed a comment documenting
round4 at the same point in lang.ts. Both sides are wanted and unrelated,
so this is a union: nonLatinCarries goes first because analyseText calls
it, and #1330's comment stays attached to the round4 it explains.

NON_LATIN_FRACTION, NON_LATIN_MIN_FRACTION and NON_LATIN_MIN_LETTERS were
already defined above. tsc --noEmit is clean and the laya-ts suite passes
1247 tests across 44 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants