Skip to content

Conversation

@andrewlock
Copy link
Member

@andrewlock andrewlock commented Nov 17, 2022

Summary of changes

  • Update build to use .NET 7 SDK
  • Update integrations to support .NET 7 runtime

Reason for change

We want to support the latest version of .NET. Previously some integrations (notably for types in the BCL) were not instrumented.

Implementation details

  • Updated the following integrations, which were broken in the .NET 7 upgrade:
    • HttpClientHandler/SocketsHttpHandler
    • Microsoft.Extensions.Logging.ILogger
    • WebRequest
    • ADO.NET "derived" integrations
    • AppSec Model binding integration
    • IAST hashing/cipher
  • Added support for some additional integrations, which weren't specifically broken, but were updated in the .NET 7 wave:
    • Npgsql 7.0
    • Microsoft.Data.Sqlite
  • Updated images to all use Windows 2022 instead of 2019. .NET 7 isn't supported on the 2019 hosted runners (as they have an older version of msbuild) so we had to update all the runners.
    • As an extension of this, the Windows Dockerfile base images had to be updated, as they have to match the host OS. We could look at continuing to run smoke tests on older images, but not critical IMO.
  • Remained building on alpine3.14 for now instead of bumping to 3.16. We can look at bumping separately if we wish
  • Added net7.0 to testing target frameworks
  • Added net7.0 to smoke tests
  • There were some weird changes to the build required
  • Updated the required version of the windows sdk to 19041 (as it's installed in the 2022 images by default) Note you need to make sure you have the Microsoft.VisualStudio.Component.Windows10SDK.19041 component installed locally to build the profiler after this change

Test coverage

There were some failing/flaky tests that I've disabled in this PR which we should investigate and re-enable ASAP

  • Profiler smoke tests log a warning: Profiler call failed with result Unspecified-Failure (80131351): pInfo->GetModuleInfo(moduleId, nullptr, 0, nullptr, nullptr, &assemblyId)
  • VersionConflict2xTests.cs: The version conflict tests require updating the NuGet we use to support .NET 7
  • DeduplicationTests.cs: The IAST deduplication tests are flaky in .NET 7 - frequently don't record any spans.
  • DefaultTransportLargePayloadTests.cs`: behaving very weirdly currently, I think this is an artifact of the new images, but not 100% sure.
  • Various debugger tests failing/flaky:
    • LineProbeEmit100SnapshotsTest
    • MethodThrowExceptionTest
    • HasLocalsAndArgumentsInGenericNestedType
  • Code coverage in .NET 461 is broken - when enabled it prevents any tests running. We've disabled it for now, but should try to reenable
  • Security tests on Windows are super flaky

Other details

This update highlighted an issue with tiered jit, which we will also need to address: #3479

@andrewlock andrewlock added type:enhancement Improvement to an existing feature area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:builds project files, build scripts, pipelines, versioning, releases, packages area:profiler Issues related to the continous-profiler labels Nov 17, 2022
@andrewlock andrewlock requested review from a team as code owners November 17, 2022 17:50
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 17, 2022

Datadog Report

Branch report: andrew/dotnet7
Commit report: 7cb7429

❄️ dd-trace-dotnet 0 Failed, 1 New Flaky, 273787 Passed, 1554 Skipped, 21m 18.73s Wall Time

New Flaky Tests (1)

  • SubmitsDataStreams - Datadog.Trace.ClrProfiler.IntegrationTests.DataStreamsMonitoringTests - Last Failure

    Expand for error
     Results do not match.
     Differences:
     Received: DataStreamsMonitoringTests.SubmitsDataStreams.received.txt
     Verified: DataStreamsMonitoringTests.SubmitsDataStreams.verified.txt
     Received Content:
     {
       Env: integration_tests,
       Service: Samples.DataStreams.Kafka,
       TracerVersion: 2.x.x.x,
       Lang: dotnet,
     ...
    

Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me, a couple of comments, but also the build is failing

@andrewlock andrewlock force-pushed the andrew/dotnet7 branch 9 times, most recently from eb12809 to ce59b09 Compare November 18, 2022 13:46
@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

Copy link
Collaborator

@OmerRaviv OmerRaviv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

.SetTargets("BuildDependencyLibs")
);

// The live debugger dependency lib _sometimes_ has to be x86/x64, and _sometimes_ has to be AnyCPU apparently
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm honestly not clear on this, in terms of why would it have to sometimes be AnyCPU (should be just x86/x64). Having said that, I'm fine with merging this as is, since @shurivich has been planning to work on this in the near term and refactor the way this dependency works, so if this works, great - we can just leave it as-is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something has changed in the way the .NET SDK uses the target platform supplied in build/test. One of the other changes I had to make is removing the `SetTagetFramework()" from the integration tests, and running those as "AnyCPU", though we still run against x64/x86, and the samples are built against x64/x86. The whole thing is a bit of a mess tbh, not just the debugger side 😬 I'm trying to do some cleanup in a separate PR but it's tough...

Copy link
Member

@tonyredondo tonyredondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Do we have plans to re-enable those flaky tests in .NET 7.0?

#if NET7_0_OR_GREATER
[SkippableTheory(Skip = "Flaky in .NET 7")]
#else
[SkippableTheory(Skip = "Flaky in .NET 7")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Skip = "Flaky in .NET 7") should be removed here right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops yeah, thanks. And yes, we should prioritise re-enabling the tests, just want people to be able to do that in parallel as this PR is blocking various other work

@andrewlock

This comment has been minimized.

Copy link
Contributor

@daniel-romano-DD daniel-romano-DD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are a titan

@andrewlock

This comment has been minimized.

Copy link
Contributor

@GreenMatan GreenMatan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great job! 💯

andrewlock and others added 20 commits November 23, 2022 13:51
They moved assembly in .NET
It uses bash, which isn't available on the new runners (We could add it) but it isn't necessary to be a separate stage anyway
There's some weird stuff going on here - adding x64 in the build integration tests seems to break some links
Also workaround some debugger weirdness (that I really don't understand), again related to paths with x64/x86
Same for the profiler integration tests, specifying the target messes things up
- The version conflict tests require updating the NuGet we use to support .NET
- The debugger tests require revisiting, as many fail. Skipping to unblock
- The IAST deduplication tests are flaky in .NET 7 - frequently don't record any spans. Skipping to unblock
- DefaultLargeTransport tests are behaving very weirdly currently, I think this is an artifact of the new images, but not 100% sure
These are flaky right now as they're timing out, but it's not clear why. We'll refactor in the longer term, but for now hoping this will help
…l, due to a bug in coverlet/the testing sdk. We should reenable at a later point
@andrewlock
Copy link
Member Author

Benchmarks Report 🐌

Benchmarks for #3482 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 753μs 574ns 2.15μs 0.379 0 0 3.22 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 544μs 303ns 1.17μs 0 0 0 2.63 KB
#3482 WriteAndFlushEnrichedTraces net472 762μs 675ns 2.53μs 0.377 0 0 3.22 KB
#3482 WriteAndFlushEnrichedTraces netcoreapp3.1 540μs 367ns 1.37μs 0 0 0 2.63 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 181ns 0.054ns 0.195ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 240ns 0.169ns 0.655ns 0.00581 0 0 424 B
master AllCycleMoreComplexBody net472 181ns 0.0982ns 0.354ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 239ns 0.247ns 0.956ns 0.0055 0 0 400 B
master BodyExtractorSimpleBody net472 292ns 0.518ns 2.01ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 248ns 0.184ns 0.688ns 0.00371 0 0 272 B
master BodyExtractorMoreComplexBody net472 15.9μs 16.4ns 63.5ns 1.2 0.0158 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.8μs 6.09ns 22.8ns 0.0899 0 0 6.75 KB
#3482 AllCycleSimpleBody net472 182ns 0.0791ns 0.296ns 0.0676 0 0 425 B
#3482 AllCycleSimpleBody netcoreapp3.1 238ns 0.3ns 1.16ns 0.00577 0 0 424 B
#3482 AllCycleMoreComplexBody net472 181ns 0.0305ns 0.118ns 0.0637 0 0 401 B
#3482 AllCycleMoreComplexBody netcoreapp3.1 238ns 0.164ns 0.636ns 0.00541 0 0 400 B
#3482 BodyExtractorSimpleBody net472 294ns 0.29ns 1.09ns 0.0574 0 0 361 B
#3482 BodyExtractorSimpleBody netcoreapp3.1 234ns 0.123ns 0.476ns 0.00361 0 0 272 B
#3482 BodyExtractorMoreComplexBody net472 16.2μs 7.26ns 27.1ns 1.21 0.0162 0 7.62 KB
#3482 BodyExtractorMoreComplexBody netcoreapp3.1 13μs 6.58ns 25.5ns 0.0905 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 176μs 153ns 594ns 0.262 0 0 20.44 KB
#3482 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3482 SendRequest netcoreapp3.1 177μs 95.7ns 371ns 0.264 0 0 20.44 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.89μs 9.73ns 46.7ns 0.161 0.000928 0 1.01 KB
master ExecuteNonQuery netcoreapp3.1 1.46μs 0.917ns 3.31ns 0.0131 0 0 1 KB
#3482 ExecuteNonQuery net472 1.8μs 1.35ns 5.24ns 0.16 0.0009 0 1.01 KB
#3482 ExecuteNonQuery netcoreapp3.1 1.43μs 0.43ns 1.61ns 0.0131 0 0 1 KB
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.42μs 0.998ns 3.87ns 0.193 0 0 1.22 KB
master CallElasticsearch netcoreapp3.1 1.49μs 0.984ns 3.81ns 0.0159 0 0 1.16 KB
master CallElasticsearchAsync net472 2.56μs 0.522ns 1.95ns 0.215 0 0 1.36 KB
master CallElasticsearchAsync netcoreapp3.1 1.72μs 1.33ns 4.96ns 0.017 0 0 1.28 KB
#3482 CallElasticsearch net472 2.49μs 0.955ns 3.7ns 0.193 0 0 1.22 KB
#3482 CallElasticsearch netcoreapp3.1 1.56μs 0.75ns 2.81ns 0.0156 0 0 1.16 KB
#3482 CallElasticsearchAsync net472 2.68μs 1.98ns 7.68ns 0.215 0 0 1.36 KB
#3482 CallElasticsearchAsync netcoreapp3.1 1.64μs 1.13ns 4.39ns 0.0166 0 0 1.28 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.69μs 2.98ns 11.5ns 0.235 0 0 1.49 KB
master ExecuteAsync netcoreapp3.1 1.73μs 1.19ns 4.47ns 0.019 0 0 1.41 KB
#3482 ExecuteAsync net472 2.7μs 0.815ns 3.05ns 0.235 0 0 1.49 KB
#3482 ExecuteAsync netcoreapp3.1 1.75μs 0.641ns 2.48ns 0.0192 0 0 1.41 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.68μs 1.15ns 4.29ns 0.45 0 0 2.83 KB
master SendAsync netcoreapp3.1 3.75μs 1.64ns 5.91ns 0.0357 0 0 2.66 KB
#3482 SendAsync net472 5.82μs 1.28ns 4.78ns 0.449 0 0 2.83 KB
#3482 SendAsync netcoreapp3.1 3.6μs 1.56ns 6.06ns 0.036 0 0 2.66 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.77μs 3.56ns 13.8ns 0.298 0 0 1.88 KB
master EnrichedLog netcoreapp3.1 2.3μs 0.987ns 3.69ns 0.0253 0 0 1.91 KB
#3482 EnrichedLog net472 2.96μs 3.12ns 12.1ns 0.298 0 0 1.88 KB
#3482 EnrichedLog netcoreapp3.1 2.33μs 1.72ns 6.21ns 0.0258 0 0 1.91 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 150μs 115ns 447ns 0.744 0.223 0 4.72 KB
master EnrichedLog netcoreapp3.1 118μs 188ns 728ns 0.059 0 0 4.55 KB
#3482 EnrichedLog net472 151μs 213ns 824ns 0.749 0.225 0 4.72 KB
#3482 EnrichedLog netcoreapp3.1 118μs 190ns 736ns 0.0589 0 0 4.55 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.46μs 1.8ns 6.96ns 0.579 0.00272 0 3.65 KB
master EnrichedLog netcoreapp3.1 4.33μs 1.84ns 6.87ns 0.0541 0 0 3.98 KB
#3482 EnrichedLog net472 5.58μs 2.06ns 7.97ns 0.579 0.00277 0 3.65 KB
#3482 EnrichedLog netcoreapp3.1 4.32μs 2.43ns 9.41ns 0.0538 0 0 3.98 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.19μs 2.4ns 9.31ns 0.228 0 0 1.44 KB
master SendReceive netcoreapp3.1 1.79μs 0.543ns 1.96ns 0.0187 0 0 1.38 KB
#3482 SendReceive net472 2.3μs 2.35ns 9.1ns 0.227 0 0 1.44 KB
#3482 SendReceive netcoreapp3.1 1.79μs 1.11ns 4.16ns 0.0187 0 0 1.38 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.74μs 1.51ns 5.86ns 0.364 0 0 2.3 KB
master EnrichedLog netcoreapp3.1 4.04μs 1.47ns 5.69ns 0.0243 0 0 1.86 KB
#3482 EnrichedLog net472 4.69μs 1.47ns 5.7ns 0.364 0 0 2.3 KB
#3482 EnrichedLog netcoreapp3.1 4.12μs 1.54ns 5.76ns 0.0246 0 0 1.86 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.12μs 0.254ns 0.983ns 0.139 0 0 875 B
master StartFinishSpan netcoreapp3.1 914ns 0.59ns 2.28ns 0.0109 0 0 824 B
master StartFinishScope net472 1.37μs 0.457ns 1.71ns 0.151 0 0 955 B
master StartFinishScope netcoreapp3.1 1.11μs 0.255ns 0.918ns 0.0131 0 0 944 B
#3482 StartFinishSpan net472 1.14μs 0.482ns 1.81ns 0.139 0 0 875 B
#3482 StartFinishSpan netcoreapp3.1 939ns 0.523ns 2.02ns 0.0112 0 0 824 B
#3482 StartFinishScope net472 1.35μs 0.646ns 2.5ns 0.152 0 0 955 B
#3482 StartFinishScope netcoreapp3.1 1.17μs 0.554ns 2.07ns 0.0124 0 0 944 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.51μs 0.766ns 2.97ns 0.151 0 0 955 B
master RunOnMethodBegin netcoreapp3.1 1.06μs 0.318ns 1.19ns 0.0128 0 0 944 B
#3482 RunOnMethodBegin net472 1.5μs 0.669ns 2.5ns 0.152 0 0 955 B
#3482 RunOnMethodBegin netcoreapp3.1 1.11μs 0.366ns 1.42ns 0.0128 0 0 944 B

@andrewlock
Copy link
Member Author

Code Coverage Report 📊

⚠️ Merging #3482 into master will will decrease line coverage by 3%
⚠️ Merging #3482 into master will will decrease branch coverage by 4%
⛔ Merging #3482 into master will will increase complexity by 164

master #3482 Change
Lines 20597 / 27986 19715 / 28020
Lines % 74% 70% -3% ⚠️
Branches 11968 / 17551 11319 / 17721
Branches % 68% 64% -4% ⚠️
Complexity 18993 19157 164

View the full report for further details:

Datadog.Trace Breakdown ⚠️

master #3482 Change
Lines % 74% 70% -3% ⚠️
Branches % 68% 64% -4% ⚠️
Complexity 18993 19157 164

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNet.ExceptionHandlerExtensions_HandleAsync_Integration -100% -100% 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNet.ThreadContext_AssociateWithCurrentThread_Integration -100% -100% 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNet.ThreadContext_DisassociateFromCurrentThread_Integration -100% -100% 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNet.AspNetResourceNameHelper -100% -97% 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.NLog.DirectSubmission.Formatting.LogEntry -100% 0% ✔️ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.NLog.DirectSubmission.Formatting.NLogDatadogLogEvent -100% 0% ✔️ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNet.ApiController_ExecuteAsync_Integration -97% -88% 0 ✔️
Datadog.Trace.AspNet.SharedItems -96% -88% 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.NLog.DirectSubmission.DirectSubmissionNLogTarget -96% -62% 0 ✔️
Datadog.Trace.AppSec.ControllerContextExtensions -95% -100% 0 ✔️
...And 31 more

The following classes were added in #3482:

File Line coverage Branch coverage Complexity
Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.ILogger.DirectSubmission.LoggerFactoryConstructorNet7Integration 90% 67% 7

View the full reports for further details:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages area:profiler Issues related to the continous-profiler area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:enhancement Improvement to an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants