-
Notifications
You must be signed in to change notification settings - Fork 31.1k
[Tokenizer] Fix slow and fast serialization
#26570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit
Hold shift + click to select a range
303a82c
fix
ArthurZucker cbf179a
Merge branch 'main' of github.com:huggingface/transformers into fix-main
ArthurZucker 01e18db
last attempt
ArthurZucker 08a560a
current work
ArthurZucker 23c9513
fix forward compatibility
ArthurZucker 0ae13ed
save all special tokens
ArthurZucker d887f68
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
ArthurZucker 72ff80e
current state
ArthurZucker b7b7d13
revert additional changes
ArthurZucker 36d5303
updates
ArthurZucker ae93856
remove tokenizer.model
ArthurZucker 88ea352
add a test and the fix
ArthurZucker ca98fbd
nit
ArthurZucker 3c22fbb
revert one more break
ArthurZucker dc93d5e
fix typefield issue
ArthurZucker 00997e9
quality
ArthurZucker 6143634
more tests
ArthurZucker 907591f
fix fields for FC
ArthurZucker 5df5a83
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
ArthurZucker 66ecb9e
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
ArthurZucker 0e7bd61
more nits?
ArthurZucker 381a0ec
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
ArthurZucker bf75334
new additional changes
ArthurZucker fafbbed
how
ArthurZucker c6de7b2
some updates
ArthurZucker 9a6e750
simplify all
ArthurZucker 8c4ec2c
more nits
ArthurZucker 621ebae
revert some things to original
ArthurZucker 6a6095e
nice
ArthurZucker e0e5dea
nits
ArthurZucker 92c7754
a small hack
ArthurZucker 9fbbafe
more nits
ArthurZucker 25e2df9
ahhaha
ArthurZucker 2b18cc2
Merge branch 'main' of github.com:huggingface/transformers into fix-main
ArthurZucker 078c94e
fixup
ArthurZucker ef1e598
update
ArthurZucker 9bf12a8
make test run on ci
ArthurZucker e6d0381
use subtesting
ArthurZucker 112e4b1
update
ArthurZucker f794a91
Update .circleci/create_circleci_config.py
ArthurZucker 65aa232
updates
ArthurZucker 8ea095b
Merge branch 'fix-main' of github.com:ArthurZucker/transformers into …
ArthurZucker efc5e7b
fixup
ArthurZucker aa569b7
nits
ArthurZucker 5ad55f3
replace typo
ArthurZucker 1c22269
fix the test
ArthurZucker 3b93653
nits
ArthurZucker a2e977a
Merge branch 'main' of github.com:huggingface/transformers into fix-main
ArthurZucker 1acf2dd
update
ArthurZucker 2dde542
None max dif pls
ArthurZucker 9ebf76e
a partial fix
ArthurZucker 6d2c00e
had to revert one thing
ArthurZucker e4bcb5e
test the fast
ArthurZucker 3d4bffd
updates
ArthurZucker 8bcb345
fixup
ArthurZucker d9e5fad
and more nits
ArthurZucker fc34148
more fixes
ArthurZucker 8389094
update
ArthurZucker 78f1ac4
Oupsy :eye:
ArthurZucker 62eb816
Merge branch 'main' of github.com:huggingface/transformers into fix-main
ArthurZucker 5c1ae9c
nits
ArthurZucker df8ab6f
fix marian
ArthurZucker 677fcb2
on our way to heaven
ArthurZucker 5a3407e
Update src/transformers/models/t5/tokenization_t5.py
ArthurZucker 856a43d
fixup
ArthurZucker a3cb498
Update src/transformers/tokenization_utils_fast.py
ArthurZucker 62cf2d0
Update src/transformers/tokenization_utils_base.py
ArthurZucker fe8bba0
fix phobert
ArthurZucker be68fc2
skip some things, test more
ArthurZucker 814d978
nits
ArthurZucker f969713
fixup
ArthurZucker 56b0619
fix deberta
ArthurZucker f2a5447
update
ArthurZucker 5d7bdab
update
ArthurZucker 49dd8b2
more updates
ArthurZucker 3a03c77
skip one test
ArthurZucker 707a688
more updates
ArthurZucker bbfc382
fix camembert
ArthurZucker b6b8aed
can't test this one
ArthurZucker dac7b89
more good fixes
ArthurZucker b4ca44e
kind of a major update
ArthurZucker 5245825
fixup
ArthurZucker 0724ebf
more fixups
ArthurZucker 066854a
fix pegasus and mpnet
ArthurZucker f646ab8
remove skipped tests
ArthurZucker 53e2390
fix phoneme tokenizer if self.verbose
ArthurZucker e0a967f
fix individual models
ArthurZucker a353871
update common tests
ArthurZucker fbc4c4f
update testing files
ArthurZucker 64a6bc4
all over again
ArthurZucker 4219b32
nits
ArthurZucker 48b937a
skip test for markup lm
ArthurZucker d1a4537
fixups
ArthurZucker 60173aa
fix order of addition in fast by sorting the added tokens decoder
ArthurZucker 8402602
proper defaults for deberta
ArthurZucker d782bbd
correct default for fnet
ArthurZucker 05ab2c2
nits on add tokens, string initialized to special if special
ArthurZucker bd6c5a5
skip irrelevant herbert tests
ArthurZucker 8a267d3
main fixes
ArthurZucker 7bda15e
update test added_tokens_serialization
ArthurZucker ac75cd3
the fix for bart like models and class instanciating
ArthurZucker 640885e
update bart
ArthurZucker 45801c0
nit!
ArthurZucker 14c576f
update idefix test
ArthurZucker 2a78cf9
fix whisper!
ArthurZucker 6f28584
some fixup
ArthurZucker c12656b
fixups
ArthurZucker 8f8c3f1
revert some of the wrong chanegs
ArthurZucker de51ef7
fixup
ArthurZucker 0f0a3fe
fixup
ArthurZucker 4b693b9
Merge branch 'main' of github.com:huggingface/transformers into fix-main
ArthurZucker 4b82043
skip marian
ArthurZucker 340df3d
skip the correct tests
ArthurZucker f9fb43d
skip for tf and flax as well
ArthurZucker File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
update
- Loading branch information
commit ef1e598514e91f4615e22285db7dcfd824e2683b
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.