Skip to content

Conversation

pakohan
Copy link

@pakohan pakohan commented Dec 12, 2019

currently, DiffCleanupSemantic only works for Ascii chars. In our company we've got Chinese chars as well, so the DiffCleanupSemantic does not work there since len("新") != len([]rune("新")).

See: https://play.golang.org/p/oTXcxo0gH1R

The Python lib has the same behaviour like the fixed version.

@sergi
Copy link
Owner

sergi commented Jan 14, 2020

Hey, thanks for this PR. I'd rather use utf8.RuneCountInString(s) than len([]rune(s)), though. The former doesn't create (and discard) a slice from the string.

More details about it here here:
https://www.reddit.com/r/golang/comments/8d4eyf/utf8runecountinstring_is_13x_faster_than_lenrunes/dxk7flk/

Copy link
Owner

@sergi sergi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@sergi sergi merged commit df97e07 into sergi:master Oct 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants