iCalendar UTF8 Encoding with/without BOM

[axunonb]

RFC5545 sect. 3.4.1 says that iCal default charset is UTF8.
Additionally, applications must generate iCalendar streams in the UTF-8 charset.

Regarding the use of a Byte Order Mark (BOM) with UTF-8 there is no mention.
However, from my experience Android, Google, Apple fail for UTF8 with BOM.

Question @ical-org/maintainers
When we serialize to a stream, should we use UTF8Encoding(false) by default (encoding parameter is null), unless the user provides an explicit (other) Encoding?

[minichma]

If I understand that correctly, the BOM would only be relevant on file level, so when persisting a calendar to a stream or reading from it. But in Ical.Net I think we don't interact with files directly. When reading from arbitrary streams, I feel we shouldn't deal with BOM. And most of the cases where we deal with utf8 encoding, this is within an ICAL, e.g. when encoding an attachment. In those cases, IMHO, a BOM wouldn't make any sense at all.

So my take would be to not emit BOM and to fail when reading UTF8 in most cases. Only when loading a whole calendar or calendar collection, we might accept BOM, still I suggest not to write it on serialization.

So long story short: IMHO we should fail when reading BOM and not write BOM, except for reading whole calendars or calendar collections. In this case we might want to ignore BOM, but even there I'm not sure, maybe failing would be the better approach even there. Even there we should not write BOM.

[axunonb]

Sorry for being not clear enough: Consider an app running under .Net Framework 4.8. So it uses the netstandard2.0 assemblies of ical.net.

When serializing

new CalendarSerializer().Serialize(someCalendar, Stream, Encoding.UTF8)

the result will include a BOM.

Why?

• In .NET Core 2.x and later (.NET 5/6/7/8), Encoding.UTF8 in StreamWriter was changed to not emit a BOM by default. Here, Encoding.UTF8 is equivalent to new UTF8Encoding(false).
• In .NET Framework, Encoding.UTF8 does include a BOM by default. new UTF8Encoding(false) must be used as encoding for getting UTF8 without a BOM.

In order to avoid such gotchas, it might be an advantage for ical.net to use UTF8Encoding(false) as the default, as it will always serialize without a BOM: Make the Serialize() to Stream methods' Encoding a parameter that defaults to null, and use UTF8Encoding(false) if it is null. In any case, this should go into the xmldoc for the Encoding parameter.

[minichma]

Understand. I agree that it would be good to have consistent behavior across versions of .NET and my suggestion would be to follow the .net core approach to not emit a BOM.

[axunonb]

Fine, I'll create a PR. When reading a file using a StreamReader with Encoding.UTF8, it will correctly handle the file regardless of whether it has a BOM or not. No need to take special care and generously accept a BOM.