Skip to content

fix(ts): cut added tokens out of the text before tokenizing, as Hugging Face does - #527

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Divit-aggarwal:fix/ts-tokenizer-added-tokens
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Divit-aggarwal:fix/ts-tokenizer-added-tokens

Conversation

@Divit-aggarwal

@Divit-aggarwal Divit-aggarwal commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What

laya-ts now cuts a tokenizer's added tokens out of the text before tokenizing it, as Hugging Face
does. encodeWithData matches them longest first (tokens that are not normalized on the raw text,
then normalized ones on what is left, honouring lstrip and rstrip) and tokenizes the pieces
between them as before.

Why

parseTokenizerJson read added_tokens only to find the special ids, so these strings were
tokenized like any other text. The shipped tokenizers define many of them:

  • the English (ModernBERT) checkpoint has 23 whitespace-run tokens (2 to 24 spaces, ids 50254 to
    50276) and placeholders such as |||EMAIL_ADDRESS|||
  • the multilingual (Gemma) checkpoint has HTML tags and control tokens such as <table>, <h1>,
    <start_of_turn> and <eos>

Real ids from the checkpoint tokenizers, Hugging Face against TS on main:

Text Hugging Face TS before
a b (English) [66, 50276, 67] [66, 209, 270]
a\n b (English) [66, 187, 50276, 67] [66, 586, 270]
<table> (multilingual) [169] [968, 1785, 235313]

So double spaces, indented logs and code, and HTML in a state reached the model as different ids from
the ones the Python agent produces. TS now returns the Hugging Face ids for all three.

Measured

Against Hugging Face tokenizers 0.23.2 on the two public checkpoint tokenizer.json files, with 6,012
seeded strings (prose, punctuation, several scripts, emoji, whitespace runs; a temporary check, not
committed). AutoTokenizer and raw tokenizers return identical ids on 46 hand-written probes, so the
reference is the one the Python agent gets.

English strings differing Multilingual strings differing
main 4,598 798
this PR 45 716

All 45 English strings that still differ contain U+FEFF, which is a separate whitespace difference
(below). The multilingual strings that still differ are the byte_fallback cases, which a companion PR
covers; with both changes the multilingual corpus differs in 0 of 6,012.

I compared timings on the real tokenizers (best of 25 runs on 4,000 characters, base and branch
interleaved, twice): about 1.3 to 1.4 ms for English prose either way, and multilingual unchanged. I did
not see a difference beyond noise, and this PR makes no performance claim. The matcher is a prefix-tree
regex so it does not try every token at every position.

Tests

21 new cases in tokenizer-parity.test.ts on two synthetic tokenizers, one ByteLevel with whitespace-run,
placeholder and special tokens and one Metaspace with tag and control tokens. Expected ids are what
Hugging Face returns for the same JSON, not what this code returns. 19 fail on main and pass with this
change; the two that pass are plain cases that must not change.

How it was verified

cd laya-ts
npm test -- --run          # 22 files, 391 tests passed
npm run test:package       # 1 package test passed
cd ..
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed
python -m compileall -q laya/ tests/                                   # exit 0

bpe-complexity, bpe, tokenizer-parity and identifier-complexity are timing-sensitive or on this
code path, so I ran them 10 times: 10 of 10 passed. The two Python commands are the CI gates; no Python
was changed.

Behaviour change

A literal added-token string in a state is now one id, as in Hugging Face, so <eos> or [SEP] typed in
a request becomes the special id the Python agent would give it. Text without added tokens is unchanged.
buildSequence already replaces the mask token in question and state text before encoding.

Scope

  • TypeScript only: tokenizer.ts and its parity test
  • No public API change: TokenizerData gains an optional added field, so a hand-built one still
    type-checks (I checked that)
  • No dependencies, no Python changes
  • Parity fix only. No prediction-accuracy claim is made

Not addressed here: the English strings that differ only around U+FEFF (JS \s treats it as
whitespace, Hugging Face's does not), and byte_fallback, which is a separate mechanism.

Checklist

  • Focused on one change
  • Rebased on the latest main
  • Tests pass locally
  • No public API change

Related: #340 also edits tokenizer.ts. A companion PR adds byte_fallback to the same file; the two
touch neighbouring lines in TokenizerData and parseTokenizerJson, so whichever merges second may need
a small conflict fix.

Written with Claude Code; I've reviewed the change and re-run the verification myself.

…ng Face does

Hugging Face matches a tokenizer's added_tokens in the text before it runs the
normalizer and pre-tokenizer, so each one becomes a single id. The TypeScript
tokenizer read added_tokens only to find the special ids, and tokenized these
strings like any other text.

The ModernBERT checkpoint defines runs of two to twenty-four spaces, and a few
placeholders such as |||EMAIL_ADDRESS|||, as added tokens: "a  b" is
[a, "  ", b] in Hugging Face and [a, " ", " b"] here, so double spaces and
indentation reached the model as different ids. The Gemma checkpoint defines
HTML tags and control tokens such as <table>, <h1>, <start_of_turn> and <eos>.

Match them longest first, as HF does: tokens that are not normalized on the raw
text, then normalized ones on what is left, honoring lstrip and rstrip, and
tokenize the pieces between them as before. The matcher is a prefix-tree regex,
so a scan does not try every token at every position. TokenizerData gains an
optional field, so a hand-built one still type-checks.

Add the cases as tests against synthetic tokenizers whose expected ids come from
Hugging Face tokenizers for the same JSON.

This is a parity fix; it makes no prediction-accuracy claim.
@Divit-aggarwal Divit-aggarwal changed the title fix(ts): cut added tokens out of the text before tokenizing, as Huggi… fix(ts): cut added tokens out of the text before tokenizing, as Hugging Face does Sep 26, 2026
@NandhaKishorM
NandhaKishorM merged commit 7a75c29 into NandhaKishorM:main Sep 27, 2026
20 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @Divit-aggarwal, merged. It combined cleanly with #526's byte fallback in the tokenizer data. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants