Repository navigation
feat(go): laya-go foundation, a pure-Go tokenizer and language detection with a Python parity lane - #1145
Conversation
…ion with a Python parity lane Adds laya-go (standard library only, go 1.22): lang (the port of laya/lang.py), an internal HF tokenizer.json BPE tokenizer, and the CPython character predicates and json.dumps/repr helpers they need. Design: Unicode behaviour is compiled in from the reference instead of taken from Go, whose unicode tables follow the toolchain. scripts/gen_tables.py generates them from CPython 15.0, laya.lang and the tokenizers crate (letter/number classes, single_word word characters, lowercase, NFC/NFD/NFKC/NFKD tables), with --check. The GPT-2 pre-tokenizer is a scanner, BPE merges on ids with cached pair ranks and a heap for long pieces. Errors are returned; unsupported components wrap ErrUnsupported; all read-only state is safe for concurrent use. Where Java and Python/the crate disagree, Python wins: final sigma in str.lower(), single_word neighbours, combining class, empty-text prefixes, and a Replace regex using \s \d \w \b \p is refused rather than read as ASCII. Parity: .github/workflows/go.yml runs on every push and pull request, regenerates the goldens from Python at that commit (gen_fixtures.py gains --out DIR; default unchanged) plus laya-go's own go_extra.json, runs go test -race -json against them, and fails on any skip via scripts/check_test_json.py. gofmt, vet, staticcheck and the oldest supported Go run beside it; govulncheck is the non-advisory security gate. Evidence: 513 tests and subtests, 0 skipped, 0 failed on the english and multilingual checkpoints, including every code point of Unicode for the predicates, normalization and lowercase, and seeded fuzz against the crate for pre-tokenization, encode and decode. Nine mutations each fail named tests. Encode throughput 22 MB/s english, 28 MB/s multilingual.
go parity: gen_tables.py --check reported tables_gen.go stale on GitHub (CPython 3.12.15) and clean locally (3.12.2). Regenerating under 3.12.14 with the pinned tokenizers 0.23.2 changes exactly one line of internal/unicodedata/tables_gen.go, the header: "from CPython 3.12.2" vs "3.12.14". Zero code points differ; every table is byte-identical on both interpreters (Unicode 15.0.0 in both). The generator wrote the PATCH version into the header, so --check could only pass on the interpreter the file was generated with. The header now names CPython major.minor and the Unicode release (the thing the data is a function of, and which stays in the Version const, so a Unicode change still fails --check). --check is not weakened. 3.12.15 itself has no uv build yet, so 3.12.14 was the nearest patch reproduced. The later go-test steps found no go-test.json because the test step was skipped after the failure; check_test_json.py now says the file is missing because the go test step did not run, instead of a FileNotFoundError traceback (with a test). go static: `go-version: stable` drifted to a Go whose export data (version 5) staticcheck v0.8.1 cannot read. Every setup-go is now pinned to 1.27.1; staticcheck v0.8.1 and govulncheck v1.9.0 both build and pass on it (and on 1.26.1). The 1.22.12 oldest-Go job is unchanged. CodeQL: add `go` to the matrix, with the same pinned setup-go and autobuild. Checked with the CodeQL CLI 2.27.1: autobuild finds laya-go/go.mod, scans 20 non-test Go files, and go-security-extended plus go-security-and-quality report 0 findings.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
… a no-op
A final review of the Object decoder found two differences from the reference.
A repeated key kept both members. Python's json.loads keeps the key at its
first position with its last value, so the two read different text:
{"a":"Ich kann mich nicht anmelden bitte hilfe",
"b":"please help me with my card refund today","a":"ok"}
Python: [('a', 'ok'), ('b', 'please help ...')], is_english True
Go: both "a" members kept, IsEnglish false
The routing verdict flipped. decodeObject now overwrites a repeated key in
place.
JSON null returned "want a JSON object, got <nil>", for a top-level null and
for a null Object field. encoding/json treats null as a no-op for its own
types, and the Object now does the same.
UnmarshalJSON called directly also accepted trailing data after the object,
and an array's close ignored io.EOF. Neither is reachable through
json.Unmarshal, but both are now refused.
TestObjectUnmarshalJSONRepeatedKeyIsLastWins fails with the overwrite
removed. TestObjectUnmarshalJSONNullAndTrailingData covers the rest.
Measured with the real english, multilingual and typed-decisions checkpoints:
go1.27.1 -race 530 ran, 0 skipped, 0 failed
go1.22.12 530 ran, 0 skipped, 0 failed, go vet clean
staticcheck clean
floor --min-tests 530 exits 0, 531 exits 1
UnmarshalJSON trimmed with bytes.TrimSpace, which also strips Unicode spaces, so a direct call with U+00A0 before null was taken as null. It now trims only JSON's four white-space characters, and the test refuses that input. The doc comment now names the convention it follows: an Unmarshaler treats null as a no-op. 530 tests ran with the real checkpoints under -race, 0 skipped, 0 failed; staticcheck clean.
#1145 adds four Python scripts under laya-go/scripts/ that record the goldens its parity lane reads. The second ruff list already covers the generator scripts of the Java, .NET and TypeScript ports for the reason the comment above it gives: an undefined name in one of these fails every parity cell of that port, and nothing else in the tree reads them. The Go lane runs them the same way, so it gets the same guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged. Verified locally rather than taken from the lane, because an 11,000-line port of the tokenizer is exactly the kind of thing that can be green and still wrong. The parity suite, against the real tokenizers. With The goldens are the Python package's, not a snapshot. The lane meets the bar the other ports set. Advisory on parity and static checks for the reason the .NET and Java lanes are (the Python package is the product, a port follows it, and a required lane would block Python work until someone ports Go), non-advisory on The supply chain is the best case available.
Two maintainer follow-ups I pushed on top, so you do not need to:
Thanks for doing the CI lane in the same PR. That is the thing that decides these. |
What and why
A first slice of a Go SDK for laya:
laya-go/, standard library only (no cgo, no third-party modules,go 1.22).lang: script and language detection, the port oflaya/lang.py(DetectScript,ScriptProfile,LatinProfileOf,GuessLatinLanguage,Analyze,DetectLanguage,IsEnglish,StateText;Scriptis a string type with constants such asScriptLatinandScriptHan;Objectis an order-preserving state that decodes from JSON in document order).internal/tokenizer: Hugging Facetokenizer.json(BPE): normalizer (NFC/NFD/NFKC/NFKD, Replace, Lowercase, Prepend, Strip, Sequence), ByteLevel and Metaspace pre-tokenizers, added tokens, byte fallback, decoder chain.internal/unicodedata,internal/pyjson: the helpers those need (CPython character predicates andstr.lower();json.dumps/reprspelled as CPython spells them).pyjsonis not used bylangor the tokenizer yet; it is the prerequisite for serialising a non-string state in the next change and has its own golden, so it lands with its tests.Nothing else: no ONNX, agent or router. The Java port is the structural reference; Python is the source of truth.
Design
tokenizer.ErrUnsupported), no panics on input, no mutable globals, a loadedTokenizerand everylangfunction are safe for concurrent use (a concurrent-use test per package runs under-race: 8 goroutines on each checkpoint's tokenizer, and 4 onlang.Analyze).unicodefollows the toolchain's release, which moves. Tables are generated Go (as Java embeds them as generated source), reproducibly bylaya-go/scripts/gen_tables.py, and--checkfails when one is stale:isalpha,\w,\d,isupper,islower,isspace,isprintable, combining class,lower(), and final-sigma context.laya.lang: script ranges, per-language word lists (and their order), thresholds.tokenizerscrate (the reference Python tokenizes with): letter/number classes (identical to the Java tables, checked), word characters beside asingle_wordtoken, lowercase, and normalization (decompositions, combining class, composition pairs).Where Java and Python/the crate disagree (Python wins)
str.lower()lowercases a word-final capital sigma to U+03C2; the Java port always gives U+03C3. Go follows Python, pinned against 1,012 seeded cases.single_wordneighbours: the crate treats combining marks, joiners and_as word characters and the other-number category as not; Java'sisAlphabetic/digit test differs on all of those. Go uses a table probed from the crate over every code point.ByteLevel(add_prefix_space=true)andPrependon empty text: the crate returns nothing / the empty string; Java returns a lone space marker / the prefix.Stripdecoder accepts one code point (Java: one UTF-16 unit). AReplaceregex normalizer using\s \d \w \b \pis refused withErrUnsupported: RE2 reads those as ASCII where the reference reads Unicode.The parity lane, and how it closes the three conditions from #672
.github/workflows/go.yml, modelled ondotnet.ymland the Java tokenizer cell. Actions are pinned by full commit SHA;check_workflow_folding.pyreports all 11 workflows clean.on: push / pull_request, nopaths:filter.gen_fixtures.py --strict --out "$GOLDENS"for the shared families (lang_tables,lang_detect,python_json,tokenizer_ids,decode_text), plusgen_go_fixtures.py --out "$GOLDENS"for the one family only this port needs;LAYA_GO_FIXTURESpoints the tests at that scratch directory, so committed copies are never read in CI.gen_fixtures.pygained a minimal--out DIR(default unchanged; under--stricta family that cannot be regenerated into a fresh directory is an error, not a skip marker). Java's own check is unchanged:gen_fixtures.py --checkstill reports20 family/families match laya (3 unverified), the 3 being the ONNX-graph families it already could not reach.go testreports a skip as success, so the suite runs with-jsonandlaya-go/scripts/check_test_json.pycounts. Any skip, failure, build failure or shortfall below the floor fails the step. The floor is exact (528, no margin): the checker exits 0 at 528 and 1 at 529 on the measured run. The checker has its own unit tests (test_check_test_json.py, 10 cases, one per way it can fail), run first in the lane. Measured: withLAYA_CHECKPOINTSunset the same suite exits 0 fromgo testand exits 1 from the checker, naming 8 skipped tests.Also: the same suite on the oldest supported Go (
GOTOOLCHAIN=go1.22.12; run locally with the real checkpoints:go vetclean, 528 run, 0 skipped, 0 failed),gofmt,go vet,staticcheck(v0.8.1), andgen_tables.py --check. Parity and static jobs are advisory (continue-on-error) like the other ports; thesecurityjob (folded-comment check andgovulncheckv1.9.0) is not.Evidence
End-to-end on real checkpoints
Rig: go1.27.1 darwin/arm64, macOS 15.6.1, Apple M2 Pro; Python 3.12.2,
tokenizers0.23.2.Checkpoints from
LAYA_CHECKPOINTS(tokenizer.json, sha256 prefix, bytes): english6c8aaa9a542084f23,583,228; multilingual609d8f4c067cd39534,363,188; typed-decisions6c8aaa9a542084f23,583,228 (byte-identical to english; no golden family is recorded against it, so it adds no case here).All goldens above were regenerated from Python at this commit into a fresh directory before the run; the shared five were also byte-identical to the committed Java fixtures. "registered" is top-level
Testfunctions (go test -list); "run" counts those plus their subtests.Skips: none with the checkpoints present. With
LAYA_CHECKPOINTSunset, 10 top-level tests skip (measured), all tokenizer tests that need a tokenizer.json, each by design because they need a tokenizer.json:TestCorpusIDsMatchPython,TestDecodeMatchesPython,TestEncodeFuzzMatchesTheCrate,TestDecodeFuzzMatchesTheCrate,TestDecodeEdges,TestEncodeLimitEdges,TestHeapAndRescanMergeAgree,TestMultilingualDecodeIsLossyByContract. That is what the guard exists to fail on.Cases checked per golden family:
http.Header,url.Values, slices of maps,map[string]Object, nested slices, arrays, named strings, pointers, named byte slices) againstlang.analyse; 3,000 random byte strings againstbytes.decode('utf-8','replace'), then throughStateText/DetectScriptand, for 600 of them, both tokenizerssingle_wordflank over every code point, both sides; 1,217 strings x 10 pre-tokenizer configs; 12 normalizer specs; 700 encode strings and 900 decode id lists per checkpointA sample, Go against the Python
tokenizerson the real checkpoints (all 10 samples matched; first 8 ids shown):Language detection on real inputs, Go against
laya.lang.analyse(7 of 7 identical, including diacritic rate and non-Latin fraction):The real checkpoints use only implemented components: english and typed-decisions declare normalizer NFC, pre-tokenizer ByteLevel, decoder ByteLevel; multilingual declares normalizer Replace with a literal String pattern (space to U+2581), pre-tokenizer Metaspace, decoder Sequence(Replace String, ByteFallback, Fuse). None uses a Replace regex, so nothing hits ErrUnsupported.
Cross-SDK comparison: Python, Java, TypeScript and Go on the same inputs
One shared corpus of 2,300 strings with a fixed seed. 300 are real sentences in 20+ languages and scripts, including mixed script, code, emoji and URLs. The other 2,000 are adversarial: combining marks, ZWJ, RTL, CJK, zero-width, whitespace-only, empty, and up to 5,000 characters. All 2,300 go through each SDK on the real
englishandmultilingualtokenizers. Python is the reference: token ids come fromtokenizers.Tokenizer.from_file(...).encode(add_special_tokens=False), and language results fromlaya.lang. No SDK source was modified to run this.Agreement with Python, out of 2,300:
is_englishSpeed on the 300 real sentences, each SDK in its own runtime. Each figure is the best of 3 after 5 warm-up passes; each rep is the mean of 10 passes. Apple M2 Pro.
The TypeScript mismatches are in laya-ts, not this PR. They are reported for the record:
"\t\t\u0085"gives[623, 16643]in Python and[186, 186, 16643]in laya-ts. A 3,744-probe repro found 47 failing inputs, all with NEL in a whitespace run. The likely cause, inferred from the repro rather than read from the source, is that JavaScript's\sexcludes NEL while the Rust tokenizer's Unicode\sincludes it.is_english, 2 inputs: random Devanagari plus control characters, where Python saysFalseand laya-ts saysTrue. Python's analysis reports the script as Devanagari and laya-ts reports Latin.detectScriptalone agrees on all 2,300.The five Java/Python differences listed under Design are not triggered by this corpus on these two checkpoints. That is why Java scores 100% here. They need a lowercasing normalizer,
single_wordadded tokens, or specific combining sequences.Independent verification, run by the reviewers rather than the author
These tests were written and run separately from the suite, by three independent review passes on this branch. Their harnesses were throwaway and are not committed.
englishandmultilingualcheckpointsdecode(skip_special_tokens true and false),detect_script,analyzelanguage/undecided,guess_latin_language,is_englishandis_english(bytes)bytes.decode('utf-8', 'replace')tokenizer.jsonenglish/typed-decisions: highest id 50,367 against a limit of 267,008;multilingual: 255,999 against 1,089,536; all dense-raceReview fixes
langnow walks a state aslang.pydoes: the values of any map whatever its keys (in sorted key order, numbers numerically and strings lexically asfmtprints a map, since a Go map has no insertion order, so a verdict can differ from a Python dict of the same items when the 4000-character budget truncates; uselang.Objectfor an order-sensitive state), slices and arrays, through pointers, interfaces and named types, to six levels; anything else contributes nothing, as in Python. Pointer and interface hops are bounded (32 per container), sox = &xends and reads as nothing, and a container that contains itself ends at the depth limit as Python's does. Previouslymap[string][]string,http.Header,url.Values,[]map[string]string,map[string]lang.Object,[][]string, named strings,*stringand maps with non-string keys were silently ignored and a German sentence came back English.\xe2\x82is one,\xff\xfeis two) at the tokenizer input and inlang, shared inunicodedata.ToValid.strings.ToValidUTF8collapses runs and is not this.-race.Design-review changes (API frozen at the first tag)
Analyseis nowAnalyze.DetectScript,Share.ScriptandAnalysis.Scriptuse aScriptstring type with a constant for every script the tables produce (ScriptLatin,ScriptHan, ...,ScriptOther,ScriptUnknown), generated bygen_tables.py. The language code stays a string, documented as ISO 639-1.Analysis.LanguageUndecidedis a method (Language == ""): over every golden and every path it was always exactly that, so the field is gone.lang.MaxDepthis exported besideMaxChars;lang.Objectdecodes JSON keeping key order and nestsObject(tested, with a mutation that reverses it).StateTextstops collecting once its budget is spent (100 leaves and 100,000 leaves both take about 3.8 us),Analyzewalks the state once and is O(leaves) (documented; 100,000 leaves take about 0.54 s). Map keys of mixed dynamic types are ordered by kind, then type name, then package path, then printed value: two distinct types sharing a name used to panic with "Field index out of range"; they now order without being compared field by field. Map keys usecmpandslices. Docs now say "likefmtfor a single key type" and that a struct state reads as nothing andAnalyze("")is English withScriptUnknown.LoadFS(fsys fs.FS, dir)andParse(tokenizerJSON, configJSON).laya-go/vX.Y.Z), pure-Golangnow androuter,presets,shortlistlater, cgo only in a futureonnxpackage with a!cgostub,langnever compiles cgo, the tokenizer goes public next change,unicodedataandpyjsonstay internal.example_test.goaddsExampleAnalyze,ExampleIsEnglishandExampleObject.Mutations (each broken in turn, named tests fail, then restored)
TestDecodeMatchesPython(92 subtests),TestCorpusIDsMatchPython,TestEncodeFuzzMatchesTheCrateTestCorpusIDsMatchPython/english/nfd-decomposed,/combining-stack,TestEncodeFuzzMatchesTheCrateTestDetectionParity/margin-exactly-one-over-englishTestNormalizationMatchesTheCrateOnRandomStringsTestLowerFinalSigmaMatchesCPythonTestPreTokenizersMatchTheCrate,TestPreTokenizerCornerCases,TestCorpusIDsMatchPython,TestEncodeFuzzMatchesTheCrateTestDecodeMatchesPython,TestLossyUTF8ReplacesMaximalSubparts,TestDecodeFuzzMatchesTheCrateTestHeapAndRescanMergeAgree,TestEncodeFuzzMatchesTheCrate,TestCorpusIDsMatchPythonTestScriptRangesMatchPython,TestScriptOfMatchesPythonEverywhereTestSameNamedKeyTypesDoNotPanic(panic)ObjectJSON keys reversedTestObjectUnmarshalJSONKeepsOrder,ExampleObjectTestStateShapesMatchPythonTestPointerLoopsTerminate(stack overflow)TestMapKeysAreReadInSortedOrderTestStateShapesMatchPython(non_string_keys)TestIDsFarBeyondTheVocabularyAreRefusedTestToValidMatchesCPythonstrings.ToValidUTF8run-collapsing at the tokenizer inputTestInvalidUTF8EncodesLikePythonBenchmark
go test -bench Encode -benchtime 2s, Apple M2 Pro, best of 3 runs interleaved with the previous revision of this branch (identical within 2%): english 150-byte sentence 6.98 µs (21.5 MB/s, 33 allocs); english 4 KB 173 µs; multilingual sentence 5.38 µs; multilingual CJK 9.78 µs.Analyze, single run: 25 µs for 450 bytes of English, 30 µs for German, 7.6 µs for Chinese. Before the rank cache, direct id table for byte-level characters and the added-token second-byte filter, english was 14 MB/s.Scanners
gofmt -l: clean.go vet ./...: clean.staticcheck2026.2.1 (v0.8.1): clean.govulncheckv1.9.0: "No vulnerabilities found" (code affected by 0; the standard-library advisories it also lists for the local toolchain are in code this module does not call).check_workflow_folding.py: 11 workflows clean.Diff
git diff --numstat upstream/main...HEAD, grouped by script: 45 files, +10,937 / −3..go, non-test, non-generated)tables_gen.gox3,go_extra.json)go.modgo.yml)laya-java/scripts/gen_fixtures.py(--out)go_extra.jsonis committed like the other ports' goldens, and the lane regenerates and checks it. Java behaviour is untouched: the only edit outsidelaya-go/and the workflow is the--outoption inlaya-java/scripts/gen_fixtures.py(+9 / −3).