How to process Unicode such as 🤗 😄？

[AlongWY]

How to process Unicode such as 🤗 😄？

when I use generate_stream, I can't process emojis, they will be processed to �.

[OlivierDehaene]

Thanks for reporting!

What model are you using?

[AlongWY]

I use bloom or opt.

[lcw99]

In my case, I use Korean, and sometimes the Korean code is split into two and displayed as ��. When I look at the server logs, it is generated normally, but when streaming, it seems to be split into two bytes.

token=Token(id=404, text=' 기', logprob=-0.10021973, special=False) generated_text=None details=None
token=Token(id=1945, text='�', logprob=0.0, special=False) generated_text=None details=None
token=Token(id=3668, text='�니다', logprob=0.0, special=False) generated_text=None details=None
token=Token(id=17, text='.', logprob=-1.6757812, special=False) generated_text=None details=None
token=Token(id=462, text=' 나', logprob=-1.6699219, special=False) generated_text=None details=None
token=Token(id=272, text='는', logprob=0.0, special=False) generated_text=None details=None
token=Token(id=2572, text=' 당신', logprob=0.0, special=False) generated_text=None details=None

[OlivierDehaene]

Should be solved on the latest release (v0.5.0).