Skip to content

tokenizer : special token handling #3538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Oct 17, 2023
Merged

tokenizer : special token handling #3538

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b592c70
Rewrite special token handling from #1931
staviq Oct 8, 2023
fc634d8
shorten param name, add st verification by type
staviq Oct 10, 2023
29e6b46
use offsets instead of copy by substr
staviq Oct 11, 2023
eac5f54
Merge branch 'master' into specialtokens
staviq Oct 11, 2023
f7b1205
formatting, remove copying iterator on delete
staviq Oct 11, 2023
04ac055
Merge branch 'master' into HEAD
ggerganov Oct 12, 2023
5c6b2be
llama : normalize code-style
ggerganov Oct 12, 2023
0f1c569
swift fix
staviq Oct 12, 2023
5974d61
print pfx/sfx if verb, main: split pfx input sfx
staviq Oct 12, 2023
1c28116
dont add space when using special tokens
staviq Oct 12, 2023
fc82541
minor : comment + spacing
ggerganov Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
swift fix
  • Loading branch information
@staviq
staviq committed Oct 12, 2023
commit 0f1c569540be0618d0a34c1e2022d81d4f9b1e6f
2 changes: 1 addition & 1 deletion examples/batched.swift/Sources/main.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ llama_print_timings(context)
private func tokenize(text: String, add_bos: Bool) -> [llama_token] {
let n_tokens = text.count + (add_bos ? 1 : 0)
let tokens = UnsafeMutablePointer<llama_token>.allocate(capacity: n_tokens)
let tokenCount = llama_tokenize(model, text, Int32(text.count), tokens, Int32(n_tokens), add_bos)
let tokenCount = llama_tokenize(model, text, Int32(text.count), tokens, Int32(n_tokens), add_bos, /*special tokens*/ false)
var swiftTokens: [llama_token] = []
for i in 0 ..< tokenCount {
swiftTokens.append(tokens[Int(i)])
Expand Down