Skip to content

feat(go): laya-go foundation, a pure-Go tokenizer and language detection with a Python parity lane - #1145

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
keemsisi:feat/go-sdk-foundation
Oct 12, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
keemsisi:feat/go-sdk-foundation

Conversation

@keemsisi

@keemsisi keemsisi commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What and why

A first slice of a Go SDK for laya: laya-go/, standard library only (no cgo, no third-party modules, go 1.22).

  • lang: script and language detection, the port of laya/lang.py (DetectScript, ScriptProfile, LatinProfileOf, GuessLatinLanguage, Analyze, DetectLanguage, IsEnglish, StateText; Script is a string type with constants such as ScriptLatin and ScriptHan; Object is an order-preserving state that decodes from JSON in document order).
  • internal/tokenizer: Hugging Face tokenizer.json (BPE): normalizer (NFC/NFD/NFKC/NFKD, Replace, Lowercase, Prepend, Strip, Sequence), ByteLevel and Metaspace pre-tokenizers, added tokens, byte fallback, decoder chain.
  • internal/unicodedata, internal/pyjson: the helpers those need (CPython character predicates and str.lower(); json.dumps / repr spelled as CPython spells them). pyjson is not used by lang or the tokenizer yet; it is the prerequisite for serialising a non-string state in the next change and has its own golden, so it lands with its tests.

Nothing else: no ONNX, agent or router. The Java port is the structural reference; Python is the source of truth.

Design

  • Idiomatic and small: errors are returned (unsupported components wrap tokenizer.ErrUnsupported), no panics on input, no mutable globals, a loaded Tokenizer and every lang function are safe for concurrent use (a concurrent-use test per package runs under -race: 8 goroutines on each checkpoint's tokenizer, and 4 on lang.Analyze).
  • No regex where Java avoids one. The GPT-2 pre-tokenizer pattern is a scanner. Merging runs on ids, keeps pair ranks and recomputes only the two a merge touches, with a heap for pieces of 32+ symbols.
  • Unicode is compiled in, not taken from Go. Go's unicode follows the toolchain's release, which moves. Tables are generated Go (as Java embeds them as generated source), reproducibly by laya-go/scripts/gen_tables.py, and --check fails when one is stale:
    • CPython 15.0: isalpha, \w, \d, isupper, islower, isspace, isprintable, combining class, lower(), and final-sigma context.
    • laya.lang: script ranges, per-language word lists (and their order), thresholds.
    • the tokenizers crate (the reference Python tokenizes with): letter/number classes (identical to the Java tables, checked), word characters beside a single_word token, lowercase, and normalization (decompositions, combining class, composition pairs).

Where Java and Python/the crate disagree (Python wins)

  • Final sigma: str.lower() lowercases a word-final capital sigma to U+03C2; the Java port always gives U+03C3. Go follows Python, pinned against 1,012 seeded cases.
  • single_word neighbours: the crate treats combining marks, joiners and _ as word characters and the other-number category as not; Java's isAlphabetic/digit test differs on all of those. Go uses a table probed from the crate over every code point.
  • Combining class: 108 code points have a class in CPython 15 that differs from the crate's (for example U+07FD, U+0C3C are starters to the crate). The crate is the reference, so the table is probed from it.
  • ByteLevel(add_prefix_space=true) and Prepend on empty text: the crate returns nothing / the empty string; Java returns a lone space marker / the prefix.
  • A Strip decoder accepts one code point (Java: one UTF-16 unit). A Replace regex normalizer using \s \d \w \b \p is refused with ErrUnsupported: RE2 reads those as ASCII where the reference reads Unicode.
  • Input that is not valid UTF-8 counts as U+FFFD (Python cannot build such a string).

The parity lane, and how it closes the three conditions from #672

.github/workflows/go.yml, modelled on dotnet.yml and the Java tokenizer cell. Actions are pinned by full commit SHA; check_workflow_folding.py reports all 11 workflows clean.

  1. CI on every push: on: push / pull_request, no paths: filter.
  2. Goldens regenerated per commit: gen_fixtures.py --strict --out "$GOLDENS" for the shared families (lang_tables, lang_detect, python_json, tokenizer_ids, decode_text), plus gen_go_fixtures.py --out "$GOLDENS" for the one family only this port needs; LAYA_GO_FIXTURES points the tests at that scratch directory, so committed copies are never read in CI. gen_fixtures.py gained a minimal --out DIR (default unchanged; under --strict a family that cannot be regenerated into a fresh directory is an error, not a skip marker). Java's own check is unchanged: gen_fixtures.py --check still reports 20 family/families match laya (3 unverified), the 3 being the ONNX-graph families it already could not reach.
  3. Fail on skip: go test reports a skip as success, so the suite runs with -json and laya-go/scripts/check_test_json.py counts. Any skip, failure, build failure or shortfall below the floor fails the step. The floor is exact (528, no margin): the checker exits 0 at 528 and 1 at 529 on the measured run. The checker has its own unit tests (test_check_test_json.py, 10 cases, one per way it can fail), run first in the lane. Measured: with LAYA_CHECKPOINTS unset the same suite exits 0 from go test and exits 1 from the checker, naming 8 skipped tests.

Also: the same suite on the oldest supported Go (GOTOOLCHAIN=go1.22.12; run locally with the real checkpoints: go vet clean, 528 run, 0 skipped, 0 failed), gofmt, go vet, staticcheck (v0.8.1), and gen_tables.py --check. Parity and static jobs are advisory (continue-on-error) like the other ports; the security job (folded-comment check and govulncheck v1.9.0) is not.

Evidence

End-to-end on real checkpoints

Rig: go1.27.1 darwin/arm64, macOS 15.6.1, Apple M2 Pro; Python 3.12.2, tokenizers 0.23.2.
Checkpoints from LAYA_CHECKPOINTS (tokenizer.json, sha256 prefix, bytes): english 6c8aaa9a542084f2 3,583,228; multilingual 609d8f4c067cd395 34,363,188; typed-decisions 6c8aaa9a542084f2 3,583,228 (byte-identical to english; no golden family is recorded against it, so it adds no case here).

python laya-java/scripts/gen_fixtures.py --strict --out $G --only lang_tables.json --only lang_detect.json \
  --only python_json.json --only tokenizer_ids.json --only decode_text.json   # LAYA_FIXTURE_CHECKPOINTS=<checkpoints>
python laya-go/scripts/gen_go_fixtures.py --out $G
cd laya-go && LAYA_GO_FIXTURES=$G LAYA_CHECKPOINTS=<checkpoints> go test -race -count=1 -json ./... \
  | python scripts/check_test_json.py - --min-tests 528

All goldens above were regenerated from Python at this commit into a fresh directory before the run; the shared five were also byte-identical to the committed Java fixtures. "registered" is top-level Test functions (go test -list); "run" counts those plus their subtests.

suite registered run skipped failed
lang 27 133 0 0
internal/tokenizer 35 373 0 0
internal/unicodedata 7 14 0 0
internal/pyjson 8 8 0 0
total 77 528 0 0

Skips: none with the checkpoints present. With LAYA_CHECKPOINTS unset, 10 top-level tests skip (measured), all tokenizer tests that need a tokenizer.json, each by design because they need a tokenizer.json: TestCorpusIDsMatchPython, TestDecodeMatchesPython, TestEncodeFuzzMatchesTheCrate, TestDecodeFuzzMatchesTheCrate, TestDecodeEdges, TestEncodeLimitEdges, TestHeapAndRescanMergeAgree, TestMultilingualDecodeIsLossyByContract. That is what the guard exists to fail on.

Cases checked per golden family:

family cases
lang_detect 106 full analyses (each: state text, script, profile, Latin profile, analyse, is_english, 86 with helper-level checks), 111 rounding values; 10 Unicode predicates swept over all 1,114,112 code points by digest
lang_tables 11 languages' word lists and order, 25 scripts (ordered scan replayed over every code point), 5 word sets, 7 thresholds
tokenizer_ids 40 corpus entries x 2 checkpoints, identity, special ids, truncation at 48
decode_text 23 texts x 2 checkpoints, 403 slice decodes, special-token handling, 5 synthetic decoder shapes, mixed-token scope
go_extra: state shapes, invalid UTF-8 15 state shapes (incl. non-string keys and self-referential list and dict) spelled as 43 Go variants (counted by a throwaway test) (maps of slices, http.Header, url.Values, slices of maps, map[string]Object, nested slices, arrays, named strings, pointers, named byte slices) against lang.analyse; 3,000 random byte strings against bytes.decode('utf-8','replace'), then through StateText/DetectScript and, for 600 of them, both tokenizers
python_json 533 doubles, 11 structured values, 36 repr strings, 28 percent0, repr digest over all of Unicode, round4 groups
go_extra (Go-only) NFC/NFD/NFKC/NFKD over every code point + 1,500 random strings; Lowercase over every code point; 1,012 final-sigma strings; single_word flank over every code point, both sides; 1,217 strings x 10 pre-tokenizer configs; 12 normalizer specs; 700 encode strings and 900 decode id lists per checkpoint

A sample, Go against the Python tokenizers on the real checkpoints (all 10 samples matched; first 8 ids shown):

checkpoint text Go ids = Python ids count
english Please refund the duplicate charge from March. 7845 23005 253 21036 4179 432 3919 15 8
english 我们被收取了两次费用,请退还。 31486 41873 16127 116 27937 17492 3218 99 19
english café naïve [MASK] 😀 68 2320 860 36060 50284 49042 211 7
multilingual Please refund the duplicate charge from March. 5651 19745 573 38294 6256 774 4482 235265 8
multilingual Je n'arrive pas à me connecter ... 3893 552 235303 114789 1806 1305 682 108829 12
multilingual मुझसे मार्च में दो बार शुल्क लिया गया 39728 238385 20579 149287 236062 9192 51484 99744 12

Language detection on real inputs, Go against laya.lang.analyse (7 of 7 identical, including diacritic rate and non-Latin fraction):

input script language English
Please refund the duplicate charge from March. latin en true
Je n'arrive pas à me connecter à mon compte depuis hier. latin fr false
Ich kann mich nicht in mein Konto einloggen und das Passwort funktioniert nicht latin de false
No puedo entrar en mi cuenta y quiero que me devuelvan el dinero latin es false
Не могу войти в аккаунт, верните деньги cyrillic none false
我们被收取了两次费用,请退还。 han none false
मुझसे मार्च में दो बार शुल्क लिया गया devanagari none false

The real checkpoints use only implemented components: english and typed-decisions declare normalizer NFC, pre-tokenizer ByteLevel, decoder ByteLevel; multilingual declares normalizer Replace with a literal String pattern (space to U+2581), pre-tokenizer Metaspace, decoder Sequence(Replace String, ByteFallback, Fuse). None uses a Replace regex, so nothing hits ErrUnsupported.

Cross-SDK comparison: Python, Java, TypeScript and Go on the same inputs

One shared corpus of 2,300 strings with a fixed seed. 300 are real sentences in 20+ languages and scripts, including mixed script, code, emoji and URLs. The other 2,000 are adversarial: combining marks, ZWJ, RTL, CJK, zero-width, whitespace-only, empty, and up to 5,000 characters. All 2,300 go through each SDK on the real english and multilingual tokenizers. Python is the reference: token ids come from tokenizers.Tokenizer.from_file(...).encode(add_special_tokens=False), and language results from laya.lang. No SDK source was modified to run this.

Agreement with Python, out of 2,300:

SDK (runtime) ids, english ids, multilingual language script is_english
Go (go1.27.1) 100% 100% 100% 100% 100%
Java (Corretto 17.0.16) 100% 100% 100% 100% 100%
TypeScript (laya-ts, node v20.14.0) 98.35% (2,262) 100% 100% 100% 99.91% (2,298)

Speed on the 300 real sentences, each SDK in its own runtime. Each figure is the best of 3 after 5 warm-up passes; each rep is the mean of 10 passes. Apple M2 Pro.

SDK tokenize, english tokenize, multilingual language analysis per call
Python 3.12.2 (reference) 18.9 µs (3.6 MB/s) 12.3 µs (5.5 MB/s) 47.4 µs
Go 6.8 µs (10.1 MB/s) 3.0 µs (22.6 MB/s) 5.9 µs
Java 13.9 µs (4.9 MB/s) 21.5 µs (3.2 MB/s) 6.5 µs
TypeScript 30.8 µs (2.2 MB/s) 20.7 µs (3.3 MB/s) 13.1 µs

The TypeScript mismatches are in laya-ts, not this PR. They are reported for the record:

  • English ids, 38 of the 2,000 adversarial strings, none of the 300 real ones. 37 of the 38 contain U+0085 (NEL) inside a whitespace run, e.g. "\t\t\u0085" gives [623, 16643] in Python and [186, 186, 16643] in laya-ts. A 3,744-probe repro found 47 failing inputs, all with NEL in a whitespace run. The likely cause, inferred from the repro rather than read from the source, is that JavaScript's \s excludes NEL while the Rust tokenizer's Unicode \s includes it.
  • is_english, 2 inputs: random Devanagari plus control characters, where Python says False and laya-ts says True. Python's analysis reports the script as Devanagari and laya-ts reports Latin. detectScript alone agrees on all 2,300.

The five Java/Python differences listed under Design are not triggered by this corpus on these two checkpoints. That is why Java scores 100% here. They need a lowercasing normalizer, single_word added tokens, or specific combining sequences.

Independent verification, run by the reviewers rather than the author

These tests were written and run separately from the suite, by three independent review passes on this branch. Their harnesses were throwaway and are not committed.

what inputs result
Differential fuzz, Go against Python, on the real english and multilingual checkpoints 43,000 strings: emoji, ZWJ, combining marks, RTL/bidi controls, CJK, zero-width/BOM, control characters, special-token literals, up to ~5,000 characters, empty, whitespace-only 0 mismatches in token ids, decode (skip_special_tokens true and false), detect_script, analyze language/undecided, guess_latin_language, is_english and is_english(bytes)
Invalid UTF-8, Go against CPython bytes.decode('utf-8', 'replace') 40,031 byte strings: 20,000 random high-biased plus every edge class (UTF-8-encoded surrogates, overlongs, truncated 4-byte sequences, F4 90+, F5–FF, 5- and 6-byte forms) 0 mismatches
Malformed tokenizer.json ~800 mutations of the real files, plus negative, duplicate, missing-unk and far-beyond-vocabulary ids every one returns an error, none panics
Decode robustness 20,000 random id lists, including negative and out-of-range ids no panics
The vocabulary-id bound against the real files english / typed-decisions: highest id 50,367 against a limit of 267,008; multilingual: 255,999 against 1,089,536; all dense no real tokenizer is refused
Concurrency 8 goroutines per checkpoint encoding and decoding under -race clean; now a committed test
Pathological states self-referential pointer, interface, slice and map; maps keyed by two distinct same-named struct types no hang and no crash, after the fixes listed below

Review fixes

  • lang now walks a state as lang.py does: the values of any map whatever its keys (in sorted key order, numbers numerically and strings lexically as fmt prints a map, since a Go map has no insertion order, so a verdict can differ from a Python dict of the same items when the 4000-character budget truncates; use lang.Object for an order-sensitive state), slices and arrays, through pointers, interfaces and named types, to six levels; anything else contributes nothing, as in Python. Pointer and interface hops are bounded (32 per container), so x = &x ends and reads as nothing, and a container that contains itself ends at the depth limit as Python's does. Previously map[string][]string, http.Header, url.Values, []map[string]string, map[string]lang.Object, [][]string, named strings, *string and maps with non-string keys were silently ignored and a German sentence came back English.
  • Invalid UTF-8 becomes U+FFFD one per maximal invalid subpart (\xe2\x82 is one, \xff\xfe is two) at the tokenizer input and in lang, shared in unicodedata.ToValid. strings.ToValidUTF8 collapses runs and is not this.
  • A vocabulary id far beyond the token count (2147483647, 1e9) is refused with an error instead of panicking or allocating gigabytes. The Rust library has no such bound (it keys ids in a map), so this is a Go-side guard: the bound is 4x the token count plus 65,536, and gaps inside it, which real checkpoints have, are allowed.
  • A real concurrency test: 8 goroutines per checkpoint, encode and decode, under -race.

Design-review changes (API frozen at the first tag)

  • Analyse is now Analyze. DetectScript, Share.Script and Analysis.Script use a Script string type with a constant for every script the tables produce (ScriptLatin, ScriptHan, ..., ScriptOther, ScriptUnknown), generated by gen_tables.py. The language code stays a string, documented as ISO 639-1.
  • Analysis.LanguageUndecided is a method (Language == ""): over every golden and every path it was always exactly that, so the field is gone.
  • lang.MaxDepth is exported beside MaxChars; lang.Object decodes JSON keeping key order and nests Object (tested, with a mutation that reverses it).
  • State reading: StateText stops collecting once its budget is spent (100 leaves and 100,000 leaves both take about 3.8 us), Analyze walks the state once and is O(leaves) (documented; 100,000 leaves take about 0.54 s). Map keys of mixed dynamic types are ordered by kind, then type name, then package path, then printed value: two distinct types sharing a name used to panic with "Field index out of range"; they now order without being compared field by field. Map keys use cmp and slices. Docs now say "like fmt for a single key type" and that a struct state reads as nothing and Analyze("") is English with ScriptUnknown.
  • The tokenizer's four loaders are LoadFS(fsys fs.FS, dir) and Parse(tokenizerJSON, configJSON).
  • README has a quickstart and the layout: no root package (tags laya-go/vX.Y.Z), pure-Go lang now and router, presets, shortlist later, cgo only in a future onnx package with a !cgo stub, lang never compiles cgo, the tokenizer goes public next change, unicodedata and pyjson stay internal. example_test.go adds ExampleAnalyze, ExampleIsEnglish and ExampleObject.

Mutations (each broken in turn, named tests fail, then restored)

broken failing tests
BPE tie-break leftmost to rightmost TestDecodeMatchesPython (92 subtests), TestCorpusIDsMatchPython, TestEncodeFuzzMatchesTheCrate
normalizer skipped TestCorpusIDsMatchPython/english/nfd-decomposed, /combining-stack, TestEncodeFuzzMatchesTheCrate
detection: margin over English 2 to 1 TestDetectionParity/margin-exactly-one-over-english
canonical reordering removed TestNormalizationMatchesTheCrateOnRandomStrings
final-sigma following-letter check removed TestLowerFinalSigmaMatchesCPython
pre-tokenizer stops returning white space TestPreTokenizersMatchTheCrate, TestPreTokenizerCornerCases, TestCorpusIDsMatchPython, TestEncodeFuzzMatchesTheCrate
lossy UTF-8 per byte, not per subpart TestDecodeMatchesPython, TestLossyUTF8ReplacesMaximalSubparts, TestDecodeFuzzMatchesTheCrate
heap tie-break reversed TestHeapAndRescanMergeAgree, TestEncodeFuzzMatchesTheCrate, TestCorpusIDsMatchPython
script lookup bound off by one TestScriptRangesMatchPython, TestScriptOfMatchesPythonEverywhere
recursing into differing dynamic key types (same-named types) TestSameNamedKeyTypesDoNotPanic (panic)
Object JSON keys reversed TestObjectUnmarshalJSONKeepsOrder, ExampleObject
reflective state walk removed TestStateShapesMatchPython
pointer-hop bound removed or reset TestPointerLoopsTerminate (stack overflow)
map keys left unsorted TestMapKeysAreReadInSortedOrder
non-string map keys ignored TestStateShapesMatchPython (non_string_keys)
sparse-id bound removed TestIDsFarBeyondTheVocabularyAreRefused
invalid UTF-8 replaced per byte TestToValidMatchesCPython
strings.ToValidUTF8 run-collapsing at the tokenizer input TestInvalidUTF8EncodesLikePython

Benchmark

go test -bench Encode -benchtime 2s, Apple M2 Pro, best of 3 runs interleaved with the previous revision of this branch (identical within 2%): english 150-byte sentence 6.98 µs (21.5 MB/s, 33 allocs); english 4 KB 173 µs; multilingual sentence 5.38 µs; multilingual CJK 9.78 µs. Analyze, single run: 25 µs for 450 bytes of English, 30 µs for German, 7.6 µs for Chinese. Before the rank cache, direct id table for byte-level characters and the added-token second-byte filter, english was 14 MB/s.

Scanners

gofmt -l: clean. go vet ./...: clean. staticcheck 2026.2.1 (v0.8.1): clean. govulncheck v1.9.0: "No vulnerabilities found" (code affected by 0; the standard-library advisories it also lists for the local toolchain are in code this module does not call). check_workflow_folding.py: 11 workflows clean.

Diff

git diff --numstat upstream/main...HEAD, grouped by script: 45 files, +10,937 / −3.

part files lines added
implementation (.go, non-test, non-generated) 17 4,173
tests 16 2,485
generated tables and golden (tables_gen.go x3, go_extra.json) 4 2,999
scripts, README, go.mod 6 1,069
workflow (go.yml) 1 202
laya-java/scripts/gen_fixtures.py (--out) 1 9

go_extra.json is committed like the other ports' goldens, and the lane regenerates and checks it. Java behaviour is untouched: the only edit outside laya-go/ and the workflow is the --out option in laya-java/scripts/gen_fixtures.py (+9 / −3).

…ion with a Python parity lane

Adds laya-go (standard library only, go 1.22): lang (the port of laya/lang.py), an internal
HF tokenizer.json BPE tokenizer, and the CPython character predicates and json.dumps/repr
helpers they need.

Design: Unicode behaviour is compiled in from the reference instead of taken from Go, whose
unicode tables follow the toolchain. scripts/gen_tables.py generates them from CPython 15.0,
laya.lang and the tokenizers crate (letter/number classes, single_word word characters,
lowercase, NFC/NFD/NFKC/NFKD tables), with --check. The GPT-2 pre-tokenizer is a scanner, BPE
merges on ids with cached pair ranks and a heap for long pieces. Errors are returned; unsupported
components wrap ErrUnsupported; all read-only state is safe for concurrent use.

Where Java and Python/the crate disagree, Python wins: final sigma in str.lower(), single_word
neighbours, combining class, empty-text prefixes, and a Replace regex using \s \d \w \b \p is
refused rather than read as ASCII.

Parity: .github/workflows/go.yml runs on every push and pull request, regenerates the goldens
from Python at that commit (gen_fixtures.py gains --out DIR; default unchanged) plus laya-go's
own go_extra.json, runs go test -race -json against them, and fails on any skip via
scripts/check_test_json.py. gofmt, vet, staticcheck and the oldest supported Go run beside it;
govulncheck is the non-advisory security gate.

Evidence: 513 tests and subtests, 0 skipped, 0 failed on the english and multilingual
checkpoints, including every code point of Unicode for the predicates, normalization and
lowercase, and seeded fuzz against the crate for pre-tokenization, encode and decode. Nine
mutations each fail named tests. Encode throughput 22 MB/s english, 28 MB/s multilingual.
go parity: gen_tables.py --check reported tables_gen.go stale on GitHub
(CPython 3.12.15) and clean locally (3.12.2). Regenerating under 3.12.14
with the pinned tokenizers 0.23.2 changes exactly one line of
internal/unicodedata/tables_gen.go, the header: "from CPython 3.12.2" vs
"3.12.14". Zero code points differ; every table is byte-identical on both
interpreters (Unicode 15.0.0 in both). The generator wrote the PATCH
version into the header, so --check could only pass on the interpreter the
file was generated with. The header now names CPython major.minor and the
Unicode release (the thing the data is a function of, and which stays in
the Version const, so a Unicode change still fails --check). --check is
not weakened. 3.12.15 itself has no uv build yet, so 3.12.14 was the
nearest patch reproduced.

The later go-test steps found no go-test.json because the test step was
skipped after the failure; check_test_json.py now says the file is missing
because the go test step did not run, instead of a FileNotFoundError
traceback (with a test).

go static: `go-version: stable` drifted to a Go whose export data
(version 5) staticcheck v0.8.1 cannot read. Every setup-go is now pinned
to 1.27.1; staticcheck v0.8.1 and govulncheck v1.9.0 both build and pass
on it (and on 1.26.1). The 1.22.12 oldest-Go job is unchanged.

CodeQL: add `go` to the matrix, with the same pinned setup-go and
autobuild. Checked with the CodeQL CLI 2.27.1: autobuild finds
laya-go/go.mod, scans 20 non-test Go files, and go-security-extended plus
go-security-and-quality report 0 findings.
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

… a no-op

A final review of the Object decoder found two differences from the reference.

A repeated key kept both members. Python's json.loads keeps the key at its
first position with its last value, so the two read different text:

    {"a":"Ich kann mich nicht anmelden bitte hilfe",
     "b":"please help me with my card refund today","a":"ok"}

    Python: [('a', 'ok'), ('b', 'please help ...')], is_english True
    Go:     both "a" members kept, IsEnglish false

The routing verdict flipped. decodeObject now overwrites a repeated key in
place.

JSON null returned "want a JSON object, got <nil>", for a top-level null and
for a null Object field. encoding/json treats null as a no-op for its own
types, and the Object now does the same.

UnmarshalJSON called directly also accepted trailing data after the object,
and an array's close ignored io.EOF. Neither is reachable through
json.Unmarshal, but both are now refused.

TestObjectUnmarshalJSONRepeatedKeyIsLastWins fails with the overwrite
removed. TestObjectUnmarshalJSONNullAndTrailingData covers the rest.
Measured with the real english, multilingual and typed-decisions checkpoints:

    go1.27.1 -race   530 ran, 0 skipped, 0 failed
    go1.22.12        530 ran, 0 skipped, 0 failed, go vet clean
    staticcheck      clean
    floor            --min-tests 530 exits 0, 531 exits 1
UnmarshalJSON trimmed with bytes.TrimSpace, which also strips Unicode spaces,
so a direct call with U+00A0 before null was taken as null. It now trims only
JSON's four white-space characters, and the test refuses that input. The doc
comment now names the convention it follows: an Unmarshaler treats null as a
no-op.

530 tests ran with the real checkpoints under -race, 0 skipped, 0 failed;
staticcheck clean.
@NandhaKishorM
NandhaKishorM merged commit 1ef1264 into NandhaKishorM:main Oct 12, 2026
42 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 12, 2026
#1145 adds four Python scripts under laya-go/scripts/ that record the goldens its parity
lane reads. The second ruff list already covers the generator scripts of the Java, .NET
and TypeScript ports for the reason the comment above it gives: an undefined name in one
of these fails every parity cell of that port, and nothing else in the tree reads them.
The Go lane runs them the same way, so it gets the same guard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. Verified locally rather than taken from the lane, because an 11,000-line port of the tokenizer is exactly the kind of thing that can be green and still wrong.

The parity suite, against the real tokenizers. With LAYA_CHECKPOINTS pointed at the shipped english and multilingual checkpoints: 530 ran, 530 passed, 0 skipped, 0 failed, which is exactly your --min-tests 530 floor. Without the checkpoints it is 256 ran, 246 passed, 10 skipped and check_test_json.py fails the run on the skips, so the fail-on-skip design does what the header says it does.

The goldens are the Python package's, not a snapshot. gen_tables.py --check matches both compiled table files, and gen_go_fixtures.py --check reproduces the committed go_extra.json byte for byte. One note for anyone who reruns that check: it reports stale under CPython 3.11, because the fixture records cpython_unicode 15.0.0 and 3.11 ships 14.0.0. The sections that move are exactly the Unicode-dependent ones (normalize, normalizers, pretokenize, python_lower, and the two fuzz families); invalid_utf8, lowercase, single_word and state_shapes are identical across both. Under 3.12, which the lane pins, it matches. Worth a line in the README next to the seed.

The lane meets the bar the other ports set. Advisory on parity and static checks for the reason the .NET and Java lanes are (the Python package is the product, a port follows it, and a required lane would block Python work until someone ports Go), non-advisory on govulncheck for the reason the JVM one is not. No paths: filter, goldens regenerated per commit, fail on skip with a measured floor. All actions SHA-pinned, and Go, staticcheck and govulncheck pinned together with the comment explaining why they move as a set.

The supply chain is the best case available. go.mod declares no requirements and there is no go.sum, so the module resolves nothing. The whole non-test import surface is the standard library and only its pure-compute and filesystem-read parts: bytes cmp container/heap encoding/json errors fmt io io/fs math math/big os path path/filepath reflect regexp runtime slices sort strconv strings unicode/utf8. No os/exec, no net, no syscall. The one net/http is http.Header used as a map[string][]string test shape in lang/shapes_test.go.

gofmt -l clean, go vet clean on Go 1.26.5 here.

Two maintainer follow-ups I pushed on top, so you do not need to:

  • laya-go/scripts/ is now in CI's second ruff list with the Java, .NET and TypeScript generator scripts. An undefined name in a fixture recorder fails every parity cell of its port and nothing else reads it, which is why that list exists.
  • Nothing else. security.yml's new go CodeQL cell is left as you wrote it, with the pinned toolchain; I will watch its first run on main.

Thanks for doing the CI lane in the same PR. That is the thing that decides these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants