⚡ refactor: Replace tiktoken with ai-tokenizer

[danny-avila]

Summary

Replaced the WASM-based tiktoken dependency with ai-tokenizer, a pure-JS tokenizer that is 5–7x faster, removes the native binary requirement, and enables first-class Claude model tokenization support.

• Rewrote the Tokenizer singleton in packages/api/src/utils/tokenizer.ts to use ai-tokenizer's count() API, backed by a two-entry encodingData map covering o200k_base and claude encodings.
• Removed freeAndResetAllEncoders, resetTokenizersIfNecessary, and tokenizerCallsCount — memory management lifecycle methods that were required by tiktoken's WASM allocator but are unnecessary for a pure-JS tokenizer.
• Added nested try/catch in getTokenCount so a double failure during error recovery logs and returns 0 rather than propagating an uncaught exception into buildMessages.
• Added Claude model detection to AgentClient.getEncoding(), selecting the claude encoding for any model whose name includes "claude" and defaulting to o200k_base for all others.
• Exported EncodingName as a public type from @librechat/api via packages/api/src/utils/index.ts for typed downstream consumers.
• Updated countTokens() to use o200k_base instead of the legacy cl100k_base encoding, and updated its JSDoc to reflect the encoding used.
• Removed the tiktokenModels export from tokens.ts, confirmed as dead code with no callers anywhere in the codebase.
• Removed stale jest.mock('tiktoken') blocks from DALLE3.spec.js, DALLE3-proxy.spec.js, and samlStrategy.spec.js.
• Updated all tests in tokenizer.spec.ts and text.spec.ts to use the new count()-based API, replacing tiktoken-specific assertions (encode/free/reset) with encoding-agnostic equivalents.

Change Type

• New feature (non-breaking change which adds functionality)

Testing

Unit tests were rewritten alongside the implementation and cover the following:

• getTokenizer returns a valid, cached AiTokenizer instance for both o200k_base and claude encodings, and defaults correctly when no encoding is specified.
• getTokenCount returns a positive count for both encodings against a real text fixture ("Hello, world!").
• Error recovery: getTokenizer is mocked to return a broken .count on the first call; the test asserts that getTokenCount logs the error, evicts the cache entry, and returns a valid non-zero count from the recreated tokenizer.
• Singleton identity is asserted via dynamic re-import.
• Integration benchmarks in text.spec.ts validate that processTextWithTokenLimit achieves ≥70% reduction in tokenizer call count versus the binary-search baseline, using the real ai-tokenizer implementation against 15k, 50k, and 120k token text fixtures.

Test Configuration:

cd packages/api && npx jest tokenizer
cd packages/api && npx jest text.spec

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in any complex areas of my code
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes

[Copilot]

Pull request overview

This PR aims to swap tokenization dependencies in the API packages by adding ai-tokenizer and removing tiktoken, presumably to standardize/streamline token counting across the codebase.

Changes:

• Add ai-tokenizer@^1.0.6 to api and packages/api.
• Remove tiktoken@^1.0.15 from those packages.
• Update root package-lock.json accordingly (adds ai-tokenizer, removes tiktoken entry).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.

File	Description
packages/api/package.json	Adds ai-tokenizer and removes tiktoken dependency for packages/api.
api/package.json	Adds ai-tokenizer and removes tiktoken dependency for api.
package-lock.json	Locks ai-tokenizer@1.0.6 and removes tiktoken from the resolved dependency tree.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.