Repository navigation
fix(ts): cut added tokens out of the text before tokenizing, as Hugging Face does - #527
Merged
NandhaKishorM merged 1 commit intoSep 27, 2026
Conversation
…ng Face does Hugging Face matches a tokenizer's added_tokens in the text before it runs the normalizer and pre-tokenizer, so each one becomes a single id. The TypeScript tokenizer read added_tokens only to find the special ids, and tokenized these strings like any other text. The ModernBERT checkpoint defines runs of two to twenty-four spaces, and a few placeholders such as |||EMAIL_ADDRESS|||, as added tokens: "a b" is [a, " ", b] in Hugging Face and [a, " ", " b"] here, so double spaces and indentation reached the model as different ids. The Gemma checkpoint defines HTML tags and control tokens such as <table>, <h1>, <start_of_turn> and <eos>. Match them longest first, as HF does: tokens that are not normalized on the raw text, then normalized ones on what is left, honoring lstrip and rstrip, and tokenize the pieces between them as before. The matcher is a prefix-tree regex, so a scan does not try every token at every position. TokenizerData gains an optional field, so a hand-built one still type-checks. Add the cases as tests against synthetic tokenizers whose expected ids come from Hugging Face tokenizers for the same JSON. This is a parity fix; it makes no prediction-accuracy claim.
Owner
|
Thanks @Divit-aggarwal, merged. It combined cleanly with #526's byte fallback in the tokenizer data. It ships in 0.3.21. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya-tsnow cuts a tokenizer's added tokens out of the text before tokenizing it, as Hugging Facedoes.
encodeWithDatamatches them longest first (tokens that are notnormalizedon the raw text,then
normalizedones on what is left, honouringlstripandrstrip) and tokenizes the piecesbetween them as before.
Why
parseTokenizerJsonreadadded_tokensonly to find the special ids, so these strings weretokenized like any other text. The shipped tokenizers define many of them:
50276) and placeholders such as
|||EMAIL_ADDRESS|||<table>,<h1>,<start_of_turn>and<eos>Real ids from the checkpoint tokenizers, Hugging Face against TS on
main:a b(English)[66, 50276, 67][66, 209, 270]a\n b(English)[66, 187, 50276, 67][66, 586, 270]<table>(multilingual)[169][968, 1785, 235313]So double spaces, indented logs and code, and HTML in a state reached the model as different ids from
the ones the Python agent produces. TS now returns the Hugging Face ids for all three.
Measured
Against Hugging Face
tokenizers0.23.2 on the two public checkpointtokenizer.jsonfiles, with 6,012seeded strings (prose, punctuation, several scripts, emoji, whitespace runs; a temporary check, not
committed).
AutoTokenizerand rawtokenizersreturn identical ids on 46 hand-written probes, so thereference is the one the Python agent gets.
mainAll 45 English strings that still differ contain U+FEFF, which is a separate whitespace difference
(below). The multilingual strings that still differ are the
byte_fallbackcases, which a companion PRcovers; with both changes the multilingual corpus differs in 0 of 6,012.
I compared timings on the real tokenizers (best of 25 runs on 4,000 characters, base and branch
interleaved, twice): about 1.3 to 1.4 ms for English prose either way, and multilingual unchanged. I did
not see a difference beyond noise, and this PR makes no performance claim. The matcher is a prefix-tree
regex so it does not try every token at every position.
Tests
21 new cases in
tokenizer-parity.test.tson two synthetic tokenizers, one ByteLevel with whitespace-run,placeholder and special tokens and one Metaspace with tag and control tokens. Expected ids are what
Hugging Face returns for the same JSON, not what this code returns. 19 fail on
mainand pass with thischange; the two that pass are plain cases that must not change.
How it was verified
bpe-complexity,bpe,tokenizer-parityandidentifier-complexityare timing-sensitive or on thiscode path, so I ran them 10 times: 10 of 10 passed. The two Python commands are the CI gates; no Python
was changed.
Behaviour change
A literal added-token string in a state is now one id, as in Hugging Face, so
<eos>or[SEP]typed ina request becomes the special id the Python agent would give it. Text without added tokens is unchanged.
buildSequencealready replaces the mask token in question and state text before encoding.Scope
tokenizer.tsand its parity testTokenizerDatagains an optionaladdedfield, so a hand-built one stilltype-checks (I checked that)
Not addressed here: the English strings that differ only around U+FEFF (JS
\streats it aswhitespace, Hugging Face's does not), and
byte_fallback, which is a separate mechanism.Checklist
mainRelated: #340 also edits
tokenizer.ts. A companion PR addsbyte_fallbackto the same file; the twotouch neighbouring lines in
TokenizerDataandparseTokenizerJson, so whichever merges second may needa small conflict fix.
Written with Claude Code; I've reviewed the change and re-run the verification myself.