Repository navigation
fix(ts): fall back to byte tokens for characters the vocab lacks, as Hugging Face does - #526
Merged
NandhaKishorM merged 1 commit intoSep 27, 2026
Conversation
…Hugging Face does The Gemma checkpoint's tokenizer sets byte_fallback: a character that is not in the vocabulary is encoded as its UTF-8 bytes, one <0xNN> token each, and only becomes <unk> if one of those byte tokens is missing too. The TypeScript metaspace encoder always emitted <unk> for such a character. The 256k vocabulary does not hold every Hangul syllable or CJK ideograph, so text in those scripts reached the model as <unk> where Hugging Face gives it the byte tokens the model was trained on. Read model.byte_fallback when parsing tokenizer.json and apply it in metaspaceEncode. The new metaspaceEncode parameter and TokenizerData field are optional and default to off, so existing callers and hand-built data behave as before. Add the cases as tests against a synthetic tokenizer whose expected ids come from Hugging Face tokenizers for the same JSON, with byte_fallback off as the control. This is a parity fix; it makes no prediction-accuracy claim.
Owner
|
Thanks @Divit-aggarwal, merged. It ships in 0.3.21. |
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya-tsnow honoursbyte_fallbackfromtokenizer.json. A character the vocabulary lacks is encodedas its UTF-8 bytes, one
<0xNN>token each, and only becomes<unk>if one of those byte tokens ismissing too.
metaspaceEncodegains an optionalbyteFallbackargument, andparseTokenizerJsonreadsmodel.byte_fallback.Why
The multilingual (Gemma) checkpoint's tokenizer sets
byte_fallback: trueand holds all but one of the256 byte tokens. The TS encoder always emitted
<unk>for a character outside the vocabulary, so textthe model was trained to read through byte tokens reached it as
<unk>.A real example from the checkpoint tokenizer:
Džis[235248, 416, 350]in Hugging Face and[235248, 3]in TS onmain.Measured
Against Hugging Face
tokenizers0.23.2 on the public multilingualtokenizer.json, encoding"a" + character + "b"for 48,048 code points (U+0020 to U+2FFF, every third of U+3000 to U+FFFF, everyseventh of U+10000 to U+2FFFF; a temporary check, not committed):
mainEvery difference on
mainwas<unk>where Hugging Face emits bytes. By block, the share of code pointsmissing from the vocabulary was 85% of Hangul syllables, 53% of CJK Unified Ideographs, over 99% of CJK
Extension A, 45% of the Arabic-to-Indic and South-East-Asian blocks, and 4% of Latin below U+0250. Those
are shares of code points, not of characters in real text, so they say where the gap is, not how often it
is hit.
On 6,012 seeded mixed-script strings the multilingual differences fall from 798 to 100. The remaining 100
are added tokens (HTML tags,
<eos>and so on), which a companion PR covers; with both changes thecorpus differs in 0 of 6,012. The English tokenizer does not use byte fallback and is unchanged.
The only missing byte token is
<0x09>(tab), which is an ordinary vocabulary character, so no real inputneeds it.
Tests
14 new cases in
tokenizer-parity.test.tson a synthetic Metaspace tokenizer with a few byte tokens:10 with
byte_fallbackon (two-byte and four-byte characters, in the middle and at the ends of words,and characters whose bytes are missing, which stay
<unk>) and 4 with it off as the control. Expected idsare what Hugging Face returns for the same JSON. 6 fail on
mainand pass with this change.How it was verified
bpe-complexity,bpe,tokenizer-parityandidentifier-complexitywere run 10 times: 10 of 10 passed.The two Python commands are the CI gates; no Python was changed.
Scope
tokenizer.tsand its parity testmetaspaceEncodeargument and theTokenizerDatafield are optional anddefault to off, so existing callers and hand-built data behave as before
for a character the vocabulary lacks
One Hugging Face detail I did not copy: when a character with no byte tokens follows a pending
<unk>,Hugging Face emits the
<unk>after later byte tokens instead of in place. It cannot occur with theshipped vocabulary, for the reason above, so
<unk>stays in place here.Checklist
mainRelated: #340 also edits
tokenizer.ts. A companion PR adds added-token matching to the same file; the twotouch neighbouring lines in
TokenizerDataandparseTokenizerJson, so whichever merges second may needa small conflict fix.
Written with Claude Code; I've reviewed the change and re-run the verification myself.