Some errors not captured if out of performance units

[bruno-garcia]

A customer on ASP.NET Core said that when they are out of performance quota, some errors are not captured.

Looking at an event, it's going through this middleware:

That's here:

sentry-dotnet/src/Sentry.AspNetCore/SentryTracingMiddleware.cs

Line 139 in 480e508

await _next(context).ConfigureAwait(false);

Since we only capture the exception further down, I wonder if we're possibly throwing an exception somewhere here:

sentry-dotnet/src/Sentry.AspNetCore/SentryTracingMiddleware.cs

Lines 149 to 210 in 480e508

	// The Transaction name was altered during the pipeline execution,
	// That could be done by user interference or by some Event Capture
	// That triggers ScopeExtensions.Populate.
	if (transaction.Name != initialName)
	{
	_options.LogDebug(
	"transaction name set from '{0}' to '{1}' during request pipeline execution.",
	initialName,
	transaction.Name);
	}
	// try to get the transaction name.
	else if (context.TryGetTransactionName() is { } transactionName &&
	!string.IsNullOrEmpty(transactionName))
	{
	_options.LogDebug(
	"Changed transaction '{0}', name set to '{1}' after request pipeline executed.",
	transaction.SpanId,
	transactionName);
	transaction.Name = transactionName;
	}
	transaction.SetExtra(OtelSemanticConventions.AttributeHttpResponseStatusCode, context.Response.StatusCode);
	var status = SpanStatusConverter.FromHttpStatusCode(context.Response.StatusCode);
	// If no Name was found for Transaction, then we don't have the route.
	if (transaction.Name == string.Empty)
	{
	var method = context.Request.Method.ToUpperInvariant();
	// If we've set a TransactionNameProvider, use that here
	var customTransactionName = context.TryGetCustomTransactionName();
	if (!string.IsNullOrEmpty(customTransactionName))
	{
	transaction.Name = $"{method} {customTransactionName}";
	((TransactionTracer)transaction).NameSource = TransactionNameSource.Custom;
	}
	else
	{
	// Finally, fallback to using the URL path.
	// e.g. "GET /pets/1"
	var path = context.Request.Path;
	transaction.Name = $"{method} {path}";
	((TransactionTracer)transaction).NameSource = TransactionNameSource.Url;
	}
	}
	if (exception is null)
	{
	transaction.Finish(status);
	}
	// Status code not yet changed to 500 but an exception does exist
	// so lets avoid passing the misleading 200 down and close only with
	// the exception instance that will be inferred as errored.
	else if (status == SpanStatus.Ok)
	{
	transaction.Finish(exception);
	}
	else
	{
	transaction.Finish(exception, status);
	}

And not reaching this:

sentry-dotnet/src/Sentry.AspNetCore/SentryTracingMiddleware.cs

Lines 213 to 216 in 480e508

	if (exception is not null)
	{
	ExceptionDispatchInfo.Capture(exception).Throw();
	}

Otherwise I'm not sure why such error would not be captured unless they are not capturing transactions (or are out of quota).

The exceptions are captured by SentryMiddleware.UnhandledException so I might be looking at it wrong as SentryMiddleware was hidden:

sentry-dotnet/src/Sentry.AspNetCore/SentryMiddleware.cs

Line 164 in 480e508

await next(context).ConfigureAwait(false);

[akonyer]

I am the one who is experiencing this issue. It seems very strange, but with no changes to the codebase at all, when we run out of transaction quota on our performance metrics, errors stop coming through. The interesting thing is that it's not ALL errors - just some. some still come through, but it is a massively reduced amount. It's like we're being throttled.

We have only used ~5% of our error reserve (15k/300k), but 100% of our performance reserve is used up.

I will get 500 errors reported from the front end (web application) and sent to sentry when an API crashes with a 500, but the back end does not sent the corresponding error to sentry.

I noticed that when I add more funds to my on-demand-budget to allow performance metrics to be captured again, then errors come through just fine. I keep having to add to this on demand reserve every day to get errors to come through until our month resets in a couple of days.

It does seem like a bug in the C# library that is somehow dropping errors when the performance logs are also dropped. And when the performance logs come through just fine the error logs do as well.

This is my config for the .net backend apps:

        builder.UseSentry(config =>
        {
            config.Dsn = dsn;
            config.Environment = serverName;
            config.MaxRequestBodySize = RequestSize.Medium;
            config.SendDefaultPii = true;
            config.IncludeActivityData = true;
            config.ServerName = serviceName;
            config.TracesSampleRate = tracesSampleRate;

            config.TracesSampler = tracesSampler;

            if (openTelemetryEnabled)
            {
                config.UseOpenTelemetry();
            }
        });

[jamescrosswell]

@bruno-garcia how/where do quotas affect the client SDKs? Is it done via Rate Limiting in the transport or is there some other mechanism?

[bruno-garcia]

@bruno-garcia how/where do quotas affect the client SDKs? Is it done via Rate Limiting in the transport or is there some other mechanism?

yeah only through X-Sentry-Rate-Limit, afaik.

@akonyer by any chance are you using openTelemetryEnabled ? Wondering if that can help narrow down.

Also, are the errors just application related? Or they seem not related to your code?

[akonyer]

No, we currently have Open Telemetry flagged off.

The errors are application code, not framework level or anything. Just crashes/exceptions in our code.

[jamescrosswell]

The interesting thing is that it's not ALL errors - just some. some still come through, but it is a massively reduced amount. It's like we're being throttled.

This seems like it could be an important clue... if it was possible to spot some pattern in those that are being dropped vs those that are not. So do the errors that you do see coming through have anything in common?

Also, this is maybe not possible in the environment where you're experiencing this but if exceptions are being dropped then that would be logged by the SentrySDK (but only if you had configured SentryOptions.Debug = true):

sentry-dotnet/src/Sentry/Http/HttpTransportBase.cs

Lines 101 to 109 in bc68b8e

	if (isRateLimited)
	{
	_options.ClientReportRecorder
	.RecordDiscardedEvent(DiscardReason.RateLimitBackoff, item.DataCategory);
	_options.LogDebug(
	"{0}: Envelope item of type {1} was discarded because it's rate-limited.",
	_typeName,
	item.TryGetType());

The presence or absence of such log messages might indicate a very different cause for the problem (whether the problem lies in the .NET SDK or in the Sentry server).