Skip to content

fix(ts): fall back to byte tokens for characters the vocab lacks, as Hugging Face does - #526

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Divit-aggarwal:fix/ts-tokenizer-byte-fallback
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Divit-aggarwal:fix/ts-tokenizer-byte-fallback

Conversation

@Divit-aggarwal

@Divit-aggarwal Divit-aggarwal commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What

laya-ts now honours byte_fallback from tokenizer.json. A character the vocabulary lacks is encoded
as its UTF-8 bytes, one <0xNN> token each, and only becomes <unk> if one of those byte tokens is
missing too. metaspaceEncode gains an optional byteFallback argument, and parseTokenizerJson reads
model.byte_fallback.

Why

The multilingual (Gemma) checkpoint's tokenizer sets byte_fallback: true and holds all but one of the
256 byte tokens. The TS encoder always emitted <unk> for a character outside the vocabulary, so text
the model was trained to read through byte tokens reached it as <unk>.

A real example from the checkpoint tokenizer: Dž is [235248, 416, 350] in Hugging Face and
[235248, 3] in TS on main.

Measured

Against Hugging Face tokenizers 0.23.2 on the public multilingual tokenizer.json, encoding
"a" + character + "b" for 48,048 code points (U+0020 to U+2FFF, every third of U+3000 to U+FFFF, every
seventh of U+10000 to U+2FFFF; a temporary check, not committed):

Characters differing
main 37,754
this PR 0

Every difference on main was <unk> where Hugging Face emits bytes. By block, the share of code points
missing from the vocabulary was 85% of Hangul syllables, 53% of CJK Unified Ideographs, over 99% of CJK
Extension A, 45% of the Arabic-to-Indic and South-East-Asian blocks, and 4% of Latin below U+0250. Those
are shares of code points, not of characters in real text, so they say where the gap is, not how often it
is hit.

On 6,012 seeded mixed-script strings the multilingual differences fall from 798 to 100. The remaining 100
are added tokens (HTML tags, <eos> and so on), which a companion PR covers; with both changes the
corpus differs in 0 of 6,012. The English tokenizer does not use byte fallback and is unchanged.

The only missing byte token is <0x09> (tab), which is an ordinary vocabulary character, so no real input
needs it.

Tests

14 new cases in tokenizer-parity.test.ts on a synthetic Metaspace tokenizer with a few byte tokens:
10 with byte_fallback on (two-byte and four-byte characters, in the middle and at the ends of words,
and characters whose bytes are missing, which stay <unk>) and 4 with it off as the control. Expected ids
are what Hugging Face returns for the same JSON. 6 fail on main and pass with this change.

How it was verified

cd laya-ts
npm test -- --run          # 22 files, 384 tests passed
npm run test:package       # 1 package test passed
cd ..
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed
python -m compileall -q laya/ tests/                                   # exit 0

bpe-complexity, bpe, tokenizer-parity and identifier-complexity were run 10 times: 10 of 10 passed.
The two Python commands are the CI gates; no Python was changed.

Scope

  • TypeScript only: tokenizer.ts and its parity test
  • No public API change: the new metaspaceEncode argument and the TokenizerData field are optional and
    default to off, so existing callers and hand-built data behave as before
  • No dependencies, no Python changes
  • Parity fix only. No prediction-accuracy claim is made, and no performance claim: the new code runs only
    for a character the vocabulary lacks

One Hugging Face detail I did not copy: when a character with no byte tokens follows a pending <unk>,
Hugging Face emits the <unk> after later byte tokens instead of in place. It cannot occur with the
shipped vocabulary, for the reason above, so <unk> stays in place here.

Checklist

  • Focused on one change
  • Rebased on the latest main
  • Tests pass locally
  • No public API change

Related: #340 also edits tokenizer.ts. A companion PR adds added-token matching to the same file; the two
touch neighbouring lines in TokenizerData and parseTokenizerJson, so whichever merges second may need
a small conflict fix.

Written with Claude Code; I've reviewed the change and re-run the verification myself.

…Hugging Face does

The Gemma checkpoint's tokenizer sets byte_fallback: a character that is not in
the vocabulary is encoded as its UTF-8 bytes, one <0xNN> token each, and only
becomes <unk> if one of those byte tokens is missing too. The TypeScript
metaspace encoder always emitted <unk> for such a character.

The 256k vocabulary does not hold every Hangul syllable or CJK ideograph, so
text in those scripts reached the model as <unk> where Hugging Face gives it
the byte tokens the model was trained on.

Read model.byte_fallback when parsing tokenizer.json and apply it in
metaspaceEncode. The new metaspaceEncode parameter and TokenizerData field are
optional and default to off, so existing callers and hand-built data behave as
before.

Add the cases as tests against a synthetic tokenizer whose expected ids come
from Hugging Face tokenizers for the same JSON, with byte_fallback off as the
control.

This is a parity fix; it makes no prediction-accuracy claim.
@Divit-aggarwal Divit-aggarwal changed the title fix(ts): fall back to byte tokens for characters the vocab lacks, as … fix(ts): fall back to byte tokens for characters the vocab lacks, as Hugging Face does Sep 26, 2026
@NandhaKishorM
NandhaKishorM merged commit b464c0e into NandhaKishorM:main Sep 27, 2026
20 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @Divit-aggarwal, merged. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants