Upgrade to Unicode 14 (to support Vithkuqi)

[pemistahl]

What version of regex are you using?

1.5.6

Describe the bug at a high level.

The letters of the Vithkuqi script, a script for writing the Albanian language, were added to Unicode version 14.0. The respective Unicode block is from U+10570 to U+105BF. I discovered that the regex \w+ does not match the letters of this block. Additionally, case-insensitive regexes starting with (?i) do not match both Vithkuqi uppercase and lowercase letters.

What are the steps to reproduce the behavior?

use regex::Regex;

let upper = "\u{10570}";          // Vithkuqi Capital Letter A
let lower = upper.to_lowercase(); // Vithkuqi Small Letter A (U+10597)

let r1 = Regex::new("(?i)^\u{10570}$").unwrap();
let r2 = Regex::new("^\\w+$").unwrap();

println!("{}", r1.is_match(upper));
println!("{}", r1.is_match(&lower));
println!("{}", r2.is_match(upper));
println!("{}", r2.is_match(&lower));

What is the actual behavior?

The actual output is:

true
false
false
false

What is the expected behavior?

The expected output is:

true
true
true
true

[BurntSushi]

If this was really added in Unicode 14, then this makes sense since I don't believe the regex crate has had its Unicode tables updated to 14 yet. They're still on 13. No real reason for it. Just hasn't been done yet.

[BurntSushi]

Also, FWIW, the regex crate does not expose any direct way to access Unicode blocks. But if Unicode's definition of word changed and new casing rules were added, then those will get automatically pulled in by updating to 14.

[pemistahl]

I see, thanks for your quick reply. Unicode 14.0 was already released in September 2021, so I think it would be a good idea to update the regex crate accordingly.

[BurntSushi]

This has been added in regex 1.6.0 on crates.io.

[pemistahl]

Awesome @BurntSushi. Thank you very much!