Logging to sentry only sent before app.Run using .NET serilog experimental logging

[KaliCZ]

Package

Sentry

.NET Flavor

.NET Core

.NET Version

9.0.301

OS

Windows

OS Version

25H2

Development Environment

Rider 2024 (Windows)

SDK Version

5.16.0

Self-Hosted Sentry Version

No response

Workload Versions

aspire 8.2.2/8.0.100 VS 17.14.36212.18

UseSentry or SentrySdk.Init call

builder.WebHost.UseSentry(options =>
        {
            options.Dsn = config.Dsn;
            options.Release = config.CommitHash;
            options.Environment = config.Environment;

            options.TracesSampleRate = 1.0d;
            options.SampleRate = 1.0f;

            options.SendDefaultPii = true; // sending info about users
            options.MaxRequestBodySize = RequestSize.Medium; // includes request body
            options.MinimumEventLevel = LogLevel.Warning;
            options.CaptureFailedRequests = false;

            options.AddExceptionFilterForType<OperationCanceledException>();
            options.SetBeforeSend(sentryEvent =>
            {
                // This happens when the client terminates a request. We can ignore this as there's not much we can about it.
                if (sentryEvent.Exception is BadHttpRequestException bre && bre.Message.Contains("Unexpected end of request content"))
                {
                    return null;
                }

                return sentryEvent;
            });

            options.UseOpenTelemetry();
        });

And also this in the serilog pipeline

    private static LoggerConfiguration LogToSentry(LoggerConfiguration loggerConfig, SentryConfiguration config)
    {
        return loggerConfig.WriteTo.Sentry(options =>
        {
            options.Dsn = config.Dsn;
            options.MinimumBreadcrumbLevel = LogEventLevel.Information;
            options.MinimumEventLevel = LogEventLevel.Warning;
            options.CaptureFailedRequests = false;

            // This doesn't work at this moment. But soon enough, it will start working.
            options.Experimental.EnableLogs = true;
            options.Experimental.SetBeforeSendLog(log =>
            {
                // TODO: Theoretically, this should already be filtered out by the logger configuration. Needs testing. Serilog wasn't supported when I wrote this.
                if (log.TryGetAttribute(ObservabilityConstants.IsSpamJobAttribute, out var isSpamJob) && isSpamJob is true)
                {
                    return log.Level >= SentryLogLevel.Warning
                        ? log
                        : null;
                }

                return log;
            });
        });
    }

Which is called this way:

        var logging = new LoggerConfiguration()
            .ReadFrom.Configuration(configForSerilogOnly)
            .Enrich.FromLogContext()
            .WriteTo.Console(outputTemplate: "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {Properties:j}{NewLine}{Exception}")
            .WriteTo.AzureApp(outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] {Message:lj}{NewLine}{Exception}");

        if (config.Observability.HoneycombApiKey is { } honeycombKey)
        {
            logging = LogToHoneycomb(logging, config, honeycombKey);
        }

        if (config.Observability.Sentry is { } sentryConfig)
        {
            logging = LogToSentry(logging, sentryConfig);
        }

        if (config.Observability.SeqApiKey is { } seqApiKey)
        {
            logging = LogToSeq(logging, serviceName: config.Observability.ServiceName, apiKey: seqApiKey);
        }

        Log.Logger = logging.CreateLogger();
        builder.Host.UseSerilog(Log.Logger);

Steps to Reproduce

• Run the application
• Observe both logging and tracing get send to all the collectors during startup
  • (sentry + seq locally, sentry + honeycomb in the cloud)
  • Mostly the EF core migrations are producing logs, but feel free to add logging to the program.cs class
• app.Run() is called and server starts
• Call some request
• Observe traces are still sent everywhere correctly
• Observe logs are no longer reaching sentry

Expected Result

Logging works even after app.run is called.

Actual Result

Logging is only sent to sentry before app.Run is called. Afterwards, the logs go to console, honeycomb, seq, but not sentry.

I've tried logging manually in Program.cs using Log.Information("21") as well as using

        using var scope2 = app.Services.CreateScope();
        var logger = scope2.ServiceProvider.GetRequiredService<ILogger<Program>>();
        logger.LogInformation("21");

And it works in the program.cs file. But not in any request, nor background service.

[linear]

NET-477

[Flash0ver]

Thank you, @KaliCZ, for using our new integration so quickly, and for the detailed report.

At first glance, this could be related to initializing 2 Hubs, a bit of a quirk of how Serilog and the Serilog SDK works in conjunction with ASP.NET Core and the ASP.NET Core SDK.

I need a bit more time to take a closer look at it later on,
but could you give it a try to also EnableLogs in the ASP.NET Core UseSentry, and not just in the Serilog WriteTo.Sentry.

Let me know if that works for you.
If it does, we probably need more docs around combining ASP.NET Core with Serilog ... I also have a off-the-top-of-my-head idea for an improvement that could make your scenario work as-is.
If it does not, I'll need to look more into it.

[KaliCZ]

@Flash0ver We're talking about the builder.WebHost.UseSentry(options => call, right? Because I'm on 5.16 and I don't see any EnableLogs or anything of that kind on the options class.

There is options.ApplySerilogScopeToEvents(); and then there is options.ExperimentalLogging, but AFAICS this only configures the min level which is Trace by default. So I'm not exactly sure what I should add.

[Flash0ver]

Oh ... I may have missed something.
You're in an .NET Aspire app ... but you are using the Sentry.AspNetCore package, right?
Then there should be

builder.WebHost.UseSentry(options =>
{
  //..
  options.Experimental.EnableLogs = true;
  options.Experimental.SetBeforeSendLog(static log => log);
  //..
}

5.16.0 has removed Experimental from SentrySdk.Logger.Log.., as we're happy with the shape of the APIs.
But we kept the Experimental on the SentryOptions (and derived) for now, as we are going to have some behavioral changes planned for 5.17.0. With (or shortly after) that, we will re-evaluate the state and plan to remove Experimental for Structured Logging entirely, to also commit to behavioral compatibility in a SemVer fashion.

Unless I dramatically misunderstand something here 🤔.