improve character category predicates

[stevengj]

As @jiahao suggested in #5576, it might be worthwhile to use utf8proc (which we are shipping with Julia anyway) to provide functions like isalnum, isalpha, iscntrl, isdigit, isgraph, islower, isprint, ispunct, isspace, isupper, and possibly isblank in string.jl. The reason is that utf8proc seems to be more up-to-date on the Unicode standard than libc, and is unhampered by legacy issues (e.g. isblank returns false for a non-breaking space, apparently for legacy reasons).

utf8proc's results are also locale-independent. This may be a plus or a minus; I don't really understand how the locale affects the results of the abovementioned predicates in libc.

[JeffBezanson]

We should definitely do this. All those predicates have canonical answers for unicode characters and shouldn't depend on locale.
One example is that the libc function isalpha(c) is defined to be equivalent to isupper(c) || islower(c), but in unicode there are letters that are neither lower- nor upper-case. The "C" locale selects the legacy behavior. So libc seems pretty bogus here.

[jiahao]

I'd be up for putting in some effort to make this happen for 0.4. Is it fair to say that things aren't so badly broken Unicode-wide that all this needs to happen in 0.3?

[stevengj]

I think that this particular issue can wait until 0.4.

[stevengj]

One difficult question is whether to even keep some of these functions, or to replace them with more Unicode-centric names and concepts.

[stevengj]

Note that these functions are all broken on Windows because the isw* functions take a single wchar_t argument, which is only 16 bits on Windows (and hence can only handle the BMP).