Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Fix Supplementary Character / Surrogate Pair info (no code changes)
Terminology and info regarding Supplementary Characters / Surrogate Pairs is either incorrect, or at least incomplete (which then leads to incorrect statements and/or code).

1. Introduce the term "Supplementary Character" since that is often what we are dealing with, not "surrogate pairs" since that is an encoding-specific concept (UTF-16 only).

2. Add comment re: Supplementary Character code point range, which helps to explain the `elif high > 0x10 then Invalid` condition (line 173).

3. Fix URI for Unicode Standard PDF, Chapter 3, and specify the name of the section (i.e. "Surrogates") instead of the section number (i.e. 3.8) since the section number was 3.7 but is now 3.8 (line 174).

4. Add comment for definition of a valid "surrogate pair" because why make the reader guess or have to go look it up when it will never change? (line 175)

5. Correct and expand comment with example long Unicode escape sequence (line 64): `"\UDEADBEEF"` is _not_ a valid escape sequence. Usage of the `\U` escape has been misstated from the very beginning, both in this documentation as well as the C# Specification documentation, and the language references for "String" for both F# and C#:
    1. `\U` is used to specify a Unicode code point (or UTF-32 code unit, which maps 1:1 with all Unicode code points, hence they are synonymous), not surrogate pairs. Hence the valid range is `00000000` - `0010FFFF`, hence the first two digits are static `0`s, and the third digit can only ever be a `0` or `1`. This escape sequence can specify either a BMP character or a Supplementary character. Supplementary characters are then encoded as a surrogate pair in UTF-16 only, not in UTF-8 or UTF-32. If you want to specify an actual surrogate pair, then use the `\u` escape, e.g. `\uD83D\uDC7D` == `\U0001F47D`.
    2. Even if you could specify a surrogate pair using `\U`, "DEADBEEF" is not valid. U+DEAD is a valid surrogate, _but_ it's a low surrogate code point and cannot be specified first in the pair (meaning, at best one could use `\UxxxxDEAD`). Also, U+BEEF is _not_ a valid surrogate code point, high or low. Surrogate code points are in the range of U+D800 to U+DFFF.

For more info, please see:

https://sqlquantumleap.com/2019/06/26/unicode-escape-sequences-across-various-languages-and-platforms-including-supplementary-characters/#fsharp
  • Loading branch information
srutzky authored Jul 12, 2019
commit 117b17a947eba1c8486e4ff67f227ad20aee6f04
7 changes: 5 additions & 2 deletions src/fsharp/lexhelp.fs
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ type lexargs =
applyLineDirectives: bool
pathMap: PathMap }

/// possible results of lexing a long unicode escape sequence in a string literal, e.g. "\UDEADBEEF"
/// possible results of lexing a long Unicode escape sequence in a string literal, e.g. "\U0001F47D",
/// "\U000000E7", or "\UDEADBEEF" returning SurrogatePair, SingleChar, or Invalid, respectively
type LongUnicodeLexResult =
| SurrogatePair of uint16 * uint16
| SingleChar of uint16
Expand Down Expand Up @@ -169,7 +170,9 @@ let unicodeGraphLong (s:string) =
if high = 0 then SingleChar(uint16 low)
// invalid encoding
elif high > 0x10 then Invalid
// valid surrogate pair - see http://www.unicode.org/unicode/uni2book/ch03.pdf, section 3.7 *)
// valid supplementary character: code points U+10000 to U+10FFFF
// valid surrogate pair: see http://www.unicode.org/versions/latest/ch03.pdf , "Surrogates" section
// high-surrogate code point (U+D800 to U+DBFF) followed by low-surrogate code point (U+DC00 to U+DFFF)
else
let codepoint = high * 0x10000 + low
let hiSurr = uint16 (0xD800 + ((codepoint - 0x10000) / 0x400))
Expand Down