Logging APIs (diff combined with John's changes)

[maryamariyan]

No description provided.

[maryamariyan]

TODO:

• Update ConsoleLoggerTest

[maryamariyan]

UPDATE:

User would instantiate logger as below:

            using var loggerFactory = LoggerFactory.Create(builder =>
            {
                builder.AddConsole(options =>
                {
                    options.Formatter = ConsoleLogFormatterNames.Json; // to choose from existing formatters 
                });
                // or simply: builder.AddConsole(ConsoleLogFormatterNames.Json);
                
                // or when using a custom formatter named 'customJson':
                builder.AddConsole("customJson")
                .AddConsoleLogFormatter<CustomConsoleLogFormatter, CustomConsoleLogFormatterOptions>((o) =>
                {
                    o.IncludeScopes = true;
                    o.ExtraCustomProperty = "property on custom formatter";
                });
            });

            var logger = loggerFactory.CreateLogger<Program>();
            var scopeObject = new { Message = "Hello" };
            using (logger.BeginScope(scopeObject))
            {
                logger.LogInformation(0, null, "all in {0}", "one");
            }

Perhaps we can make the LogFormatter implementations internal and have helper methods like such to setup the properties as the props are public:

                .AddDefaultConsoleLogFormatter(o =>
                {
                    o.DisableColors = true;
                })

Possible selections without needing to implement a custom log formatter:

Default:

Compact:

Systemd:

Json:

sample appsettings.json, as discussed we would be looking up Console:Logging:FomratterOptions to pickup options for formatters:

{
  "Logging": {
    "LogLevel": {
      "Default": "None",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    },

    "Console": {
      "LogLevel": {
        "Default": "Information"
      },
      "Formatter": "Systemd",
      "FormatterOptions": {
        "JsonWriterOptions": {
          "Indented": true
        },
        "JsonSerializerOptions": {
          "WriteIndented": true
        },
        "ExtraCustomProperty": "modified",
        "IncludeScopes": true,
        "TimestampFormat": "HH:mm:ss",
        "UseUtcTimestamp": true,
        "DisableColors": true
      }
    }
  },
  "AllowedHosts": "*"
}

Last note, I thought IConsoleLogFormatter needs to have another overload of Format<TState>... as seen below so a formatter (like you see in the screenshot for Compact) could find params in a formatted message that is not necessarily string and potentially color them differently, as currently some existing out of the box structured log formatters do, but our existing format in master branch does not support it.

    public partial interface IConsoleLogFormatter
    {
        string Name { get; }
        Microsoft.Extensions.Logging.Console.LogMessageEntry Format(Microsoft.Extensions.Logging.LogLevel logLevel, string logName, int eventId, string message, System.Exception exception, Microsoft.Extensions.Logging.IExternalScopeProvider scopeProvider);
+        Microsoft.Extensions.Logging.Console.LogMessageEntry Format<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, string logName, int eventId, TState state, System.Exception exception, System.Func<TState, System.Exception, string> formatter, Microsoft.Extensions.Logging.IExternalScopeProvider scopeProvider);
    }

To make it work I also changed LogMessageEntry like below

    public readonly partial struct LogMessageEntry
    {
        public readonly bool LogAsError;
        public readonly Microsoft.Extensions.Logging.Console.ConsoleMessage[] Messages;
        public LogMessageEntry(Microsoft.Extensions.Logging.Console.ConsoleMessage[] messages, bool logAsError = false) { throw null; }
    }
    public readonly partial struct ConsoleMessage
    {
        public readonly System.ConsoleColor? Background;
        public readonly System.ConsoleColor? Foreground;
        public readonly string Message;
        public ConsoleMessage(string message, System.ConsoleColor? background = default(System.ConsoleColor?), System.ConsoleColor? foreground = default(System.ConsoleColor?)) { throw null; }
    }

So we won't expose IConsole and there won't be a callback.
cc @eerhardt who I also chatted and got help with regarding LogMessageEntry.

The reason ConsoleMessage[] is an array in LogMessageEntry is that it helps remove fields like
MessageColor, LevelString and TimeStamp from LogLevelEntry and also, it enables different colors on a log message.

Additional detail:
This is what I presented on the previous iteration:

    public readonly partial struct LogMessageEntry
    {
        public readonly System.ConsoleColor? LevelBackground;
        public readonly System.ConsoleColor? LevelForeground;
        public readonly string LevelString;
        public readonly bool LogAsError;
        public readonly string Message;
        public readonly System.ConsoleColor? MessageColor;
        public readonly string TimeStamp;
+        public readonly System.Action<Microsoft.Extensions.Logging.Console.IConsole> WriteCallback;
+        public LogMessageEntry(string message, string timeStamp = null, string levelString = null, System.ConsoleColor? levelBackground = default(System.ConsoleColor?), System.ConsoleColor? levelForeground = default(System.ConsoleColor?), System.ConsoleColor? messageColor = default(System.ConsoleColor?), bool logAsError = false, System.Action<Microsoft.Extensions.Logging.Console.IConsole> writeCallback = null) { throw null; }
    }

Problems were: IConsole was being exposed and the control of the output thread would go to the WriteCallback so the whatever is done in the callback could add a delay to that thread.

[shirhatti]

Should we simplify the ctor and move ScopeProvider into the service container?

[maryamariyan]

Summary so far:

New APIs on existing classes:

namespace Microsoft.Extensions.Logging
{
    public static partial class ConsoleLoggerExtensions
    {
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddConsole(this Microsoft.Extensions.Logging.ILoggingBuilder builder, string formatterName) { throw null; }
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddConsoleLogFormatter<TFormatter, TOptions>(this Microsoft.Extensions.Logging.ILoggingBuilder builder) where TFormatter : class, Microsoft.Extensions.Logging.Console.IConsoleLogFormatter where TOptions : class { throw null; }
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddConsoleLogFormatter<TFormatter, TOptions>(this Microsoft.Extensions.Logging.ILoggingBuilder builder, System.Action<TOptions> configure) where TFormatter : class, Microsoft.Extensions.Logging.Console.IConsoleLogFormatter where TOptions : class { throw null; }
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddDefaultConsoleLogFormatter(this Microsoft.Extensions.Logging.ILoggingBuilder builder, System.Action<Microsoft.Extensions.Logging.Console.DefaultConsoleLogFormatterOptions> configure) { throw null; }
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddJsonConsoleLogFormatter(this Microsoft.Extensions.Logging.ILoggingBuilder builder, System.Action<Microsoft.Extensions.Logging.Console.JsonConsoleLogFormatterOptions> configure) { throw null; }
        public static Microsoft.Extensions.Logging.ILoggingBuilder AddSystemdConsoleLogFormatter(this Microsoft.Extensions.Logging.ILoggingBuilder builder, System.Action<Microsoft.Extensions.Logging.Console.SystemdConsoleLogFormatterOptions> configure) { throw null; }
    }
}

namespace Microsoft.Extensions.Logging.Console
{
    [Microsoft.Extensions.Logging.ProviderAliasAttribute("Console")]
    public partial class ConsoleLoggerProvider : Microsoft.Extensions.Logging.ILoggerProvider, Microsoft.Extensions.Logging.ISupportExternalScope, System.IDisposable
    {
        public ConsoleLoggerProvider(Microsoft.Extensions.Options.IOptionsMonitor<Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions> options, System.Collections.Generic.IEnumerable<Microsoft.Extensions.Logging.Console.IConsoleLogFormatter> formatters) { }
    }
}

Above APIs show how the setup usage could look like. Since we would be having option for different kinds of log formatters (built-in json, default and systemd), then it makes sense for each to have their own formatter options. If someone would like to have their own custom formatter and formatter options that would also be possible and they would be using AddConsoleLogFormatter API.

New class and APIs:

*FormatterOptions:

namespace Microsoft.Extensions.Logging.Console
{
    public partial class DefaultConsoleLogFormatterOptions : Microsoft.Extensions.Logging.Console.SystemdConsoleLogFormatterOptions
    {
        public DefaultConsoleLogFormatterOptions() { }
        public bool DisableColors { get { throw null; } set { } }
        public bool MultiLine { get { throw null; } set { } }
    }
    public static partial class ConsoleLogFormatterNames
    {
        public const string Colored = "default";
        public const string Json = "json";
        public const string Systemd = "systemd";
    }
    public partial class JsonConsoleLogFormatterOptions : Microsoft.Extensions.Logging.Console.SystemdConsoleLogFormatterOptions
    {
        public JsonConsoleLogFormatterOptions() { }
        public System.Text.Json.JsonWriterOptions JsonWriterOptions { get { throw null; } set { } }
    }
    public partial class SystemdConsoleLogFormatterOptions
    {
        public SystemdConsoleLogFormatterOptions() { }
        public bool IncludeScopes { get { throw null; } set { } }
        public Microsoft.Extensions.Logging.LogLevel LogToStandardErrorThreshold { get { throw null; } set { } }
        public string TimestampFormat { get { throw null; } set { } }
        public bool UseUtcTimestamp { get { throw null; } set { } }
    }
}

a custom log formatter would have to implement the following interface:

    public partial interface IConsoleLogFormatter
    {
        string Name { get; }
        Microsoft.Extensions.Logging.Console.LogMessageEntry Format<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, string logName, int eventId, TState state, System.Exception exception, System.Func<TState, System.Exception, string> formatter, Microsoft.Extensions.Logging.IExternalScopeProvider scopeProvider);
    }

Note above that the console log formatter would return a LogMessageEntry which could look like any of the options below, (each with their own drawbacks or benefits).

• Option 1: using ConsoleMessage array

    public readonly partial struct ConsoleMessage
    {
        public readonly System.ConsoleColor? Background;
        public readonly string Message;
        public readonly System.ConsoleColor? Foreground;
        public ConsoleMessage(string message, System.ConsoleColor? background = default(System.ConsoleColor?), System.ConsoleColor? foreground = default(System.ConsoleColor?)) { throw null; }
    }
    public readonly partial struct LogMessageEntry
    {
        public readonly bool LogAsError;
        public readonly Microsoft.Extensions.Logging.Console.ConsoleMessage[] Messages;
        public LogMessageEntry(Microsoft.Extensions.Logging.Console.ConsoleMessage[] messages, bool logAsError = false) { throw null; }
    }

• Option 2: using ConsoleMessage optimized with params arguments
  Similar to above but allowing params arguments to reduce allocation for Formatters that don't usually need a lot of elements in the ConsoleMessage array

    public readonly partial struct ConsoleMessage
    {
        public readonly System.ConsoleColor? Background;
        public readonly string Message;
        public readonly System.ConsoleColor? Foreground;
        public ConsoleMessage(string message, System.ConsoleColor? background = default(System.ConsoleColor?), System.ConsoleColor? foreground = default(System.ConsoleColor?)) { throw null; }
    }
    public readonly partial struct LogMessageEntry
    {
        public readonly bool LogAsError;
        public LogMessageEntry(Microsoft.Extensions.Logging.Console.ConsoleMessage message1, Microsoft.Extensions.Logging.Console.ConsoleMessage message2, Microsoft.Extensions.Logging.Console.ConsoleMessage message3, bool logAsError = false) { throw null; }
        public LogMessageEntry(Microsoft.Extensions.Logging.Console.ConsoleMessage message1, Microsoft.Extensions.Logging.Console.ConsoleMessage message2, bool logAsError = false) { throw null; }
        public LogMessageEntry(Microsoft.Extensions.Logging.Console.ConsoleMessage message1, bool logAsError = false) { throw null; }
        public LogMessageEntry(bool logAsError = false, params Microsoft.Extensions.Logging.Console.ConsoleMessage[] messages) { throw null; }
    }

• Option 3: using IConsole and WriteCallback:
  As discussed we would need to expose an IConsole interface and there would need to be a guideline for formatter implementations on how to properly use WriteCallback so as not to block the output thread.

    public readonly partial struct LogMessageEntry
    {
        public readonly System.ConsoleColor? LevelBackground;
        public readonly System.ConsoleColor? LevelForeground;
        public readonly string LevelString;
        public readonly bool LogAsError;
        public readonly string Message;
        public readonly System.ConsoleColor? MessageColor;
        public readonly string TimeStamp;
        public readonly System.Action<Microsoft.Extensions.Logging.Console.IConsole> WriteCallback;
        public LogMessageEntry(string message, string timeStamp = null, string levelString = null, System.ConsoleColor? levelBackground = default(System.ConsoleColor?), System.ConsoleColor? levelForeground = default(System.ConsoleColor?), System.ConsoleColor? messageColor = default(System.ConsoleColor?), bool logAsError = false, System.Action<Microsoft.Extensions.Logging.Console.IConsole> writeCallback = null) { throw null; }
    }
    public partial interface IConsole
    {
        void Flush();
        void Write(string message, System.ConsoleColor? background, System.ConsoleColor? foreground);
        void WriteLine(string message, System.ConsoleColor? background, System.ConsoleColor? foreground);
    }

• Option 4: LogMessage entry would have a single message string containing ANSI color codes in the message. Then we keep the console message writing internal and just parse colors and messages internally.

    public readonly partial struct LogMessageEntry
    {
        public readonly bool LogAsError;
        public readonly string Message;
        public LogMessageEntry(bool logAsError = false, string message) { throw null; }
    }

Another option that could be explored is that we introduce an interface called IConsoleMessageBuilder like below:

   public interface IConsoleMessageBuilder
   {
      IConsoleMessageBuilder SetColor(ConsoleColor color);
      IConsoleMessageBuilder Append(string message);
      string ToString();
   }

This has not been thought through much yet, but the benefit could be that when implementing the IConsoleLogFormatter.Format method, we would use IConsoleMessageBuilder similar to how the threadstatic StringBuilder is being used in ConsoleLogger to create a LogMessageEntry today here, with the difference that along the lines of doing that Append, we could use SetColor so that would make sure colors appear differently within a log message. Any of the 4 options above could satisfy this but using an IConsoleMessageBuilder this way could help with some of their shortcomings (e.g. with option 1 we have an array allocation cost)

deprecated APIs

    [System.ObsoleteAttribute("ConsoleLoggerFormat has been deprecated.", false)]
    public enum ConsoleLoggerFormat
    {
        Default = 0,
        Systemd = 1,
    }
    public partial class ConsoleLoggerOptions
    {
        [System.ObsoleteAttribute("ConsoleLoggerOptions.DisableColors has been deprecated. Please use ColoredConsoleLogFormatterOptions.DisableColors instead.", false)]
        public bool DisableColors { get { throw null; } set { } }
        [System.ObsoleteAttribute("ConsoleLoggerOptions.Format has been deprecated. Please use ConsoleLoggerOptions.FormatterName instead.", false)]
        public Microsoft.Extensions.Logging.Console.ConsoleLoggerFormat Format { get { throw null; } set { } }
        [System.ObsoleteAttribute("ConsoleLoggerOptions.IncludeScopes has been deprecated..", false)]
        public bool IncludeScopes { get { throw null; } set { } }
        [System.ObsoleteAttribute("ConsoleLoggerOptions.LogToStandardErrorThreshold has been deprecated..", false)]
        public Microsoft.Extensions.Logging.LogLevel LogToStandardErrorThreshold { get { throw null; } set { } }
        [System.ObsoleteAttribute("ConsoleLoggerOptions.TimestampFormat has been deprecated..", false)]
        public string TimestampFormat { get { throw null; } set { } }
        [System.ObsoleteAttribute("ConsoleLoggerOptions.UseUtcTimestamp has been deprecated..", false)]
        public bool UseUtcTimestamp { get { throw null; } set { } }
    }
}