Adding two variations of Japanese-BERT #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

debymf merged 4 commits into master from japanese-bert

Apr 11, 2023

+738 −79

Member

debymf commented Apr 11, 2023

No description provided.

debymf added 2 commits

April 11, 2023 13:51


          adding japanese-bert models

f21b1ca


          fixing indentation.

cb9738f

debymf requested a review from bwdGitHub

April 11, 2023 13:55

bwdGitHub reviewed

View reviewed changes

+bert/+tokenizer/+internal/FullTokenizer.m Outdated Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

+bert/+tokenizer/+internal/FullTokenizer.m Outdated Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

+bert/+tokenizer/BERTTokenizer.m Outdated Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

FineTuneBERTJapanese.m Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

bert.m Outdated Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

test/bert/tokenizer/internal/tBasicTokenizer.m Show resolved Hide resolved

bwdGitHub reviewed

View reviewed changes

bert.m Show resolved Hide resolved

debymf added 2 commits

April 11, 2023 17:28


          Adressing feedback.

80a159c


          Changing expected error msg in test.

3a2c48f

bwdGitHub reviewed

View reviewed changes

+bert/+tokenizer/+internal/FullTokenizer.m

    
                              this.Basic = bert.tokenizer.internal.BasicTokenizer('IgnoreCase',nvp.IgnoreCase);

                          else

                              assert(isa(nvp.BasicTokenizer,'bert.tokenizer.internal.Tokenizer'),"BasicTokenizer must be a bert.tokenizer.internal.Tokenizer implementation.");

                              mustBeA(nvp.BasicTokenizer,'bert.tokenizer.internal.Tokenizer');

Collaborator

bwdGitHub Apr 11, 2023

Ah, this caused a test failure since the verifyError wants the error ID.

bwdGitHub approved these changes

View reviewed changes

Collaborator

bwdGitHub left a comment

Looks good, feel free to merge when you push the fix for the test failure.

debymf merged commit 6715db0 into master

debymf deleted the japanese-bert branch

April 19, 2023 21:39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet