P99 Latency for .net 8 services increased > 10x

[jashook]

Description

It seems our applications have seen a ~9-10x difference in their gen0 min budget leading to larger than expected P99 differences between .net 6 and .net 8.

I am reticent to attach full traces below, so please some of the data for more information.

net6 MinBudeget:

GlobalHeapHistory FinalYoungestDesired="32,432,160" NumHeaps="36"

net8 MinBudget

GlobalHeapHistory FinalYoungestDesired="207,272,672" NumHeaps="36"

Overall, these services are using the same underlying hardware. I have attached an lscpu example below. Roblox runs services using cpu-shares therefore, these services are fingerprinting the whole machines; however, we are running with DOTNET_PROCESSOR_COUNT=36 for this specific service.

Average GC time has increased from ~24ms to ~200ms for this specific service after moving to .net 8.

lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  64
  On-line CPU(s) list:   0-63
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Gold 6314U CPU @ 2.30GHz
    CPU family:          6
    Model:               106
    Thread(s) per core:  2
    Core(s) per socket:  32
    Socket(s):           1
    Stepping:            6
    CPU max MHz:         3400.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4600.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art a
                         rch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca ss
                         e4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs i
                         bpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap av
                         x512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect 
                         wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx51
                         2_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   1.5 MiB (32 instances)
  L1i:                   1 MiB (32 instances)
  L2:                    40 MiB (32 instances)
  L3:                    48 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-63
Vulnerabilities:         
  Gather data sampling:  Mitigation; Microcode
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Not affected

Reproduction Steps

The description should have enough information to repro.

Expected behavior

10x larger P99 difference due to larger gc pause times.

Actual behavior

Same as expected.

Regression?

This is a regression between .net 6 and .net 8

Known Workarounds

Manually setting the gen0 min budget to the same value as .net 6 should resolve the issue.

Configuration

No response

Other information

No response

[jashook]

Related to #52623

[jashook]

cc @jeffschwMSFT

[jashook]

Also cc @Maoni0

[jashook]

Sorry for the direct tag, but this is clearly area-gc and I can no longer modify as such.

[mrsharm]

Hi @jashook, this seems like it's the same issue solved by: #100502

To confirm this, can you please send us the result of:

getconf -a | grep LEVEL4_CACHE?

The fix should be made available in the next servicing release of .NET 8.

Thanks,
M.

[jashook]

LEVEL4_CACHE_SIZE                  0
LEVEL4_CACHE_ASSOC                 
LEVEL4_CACHE_LINESIZE              

[jashook]

The fix still seems to still be a regression from the existing behavior in .net 6. As is the value read will default to the max l3 cache size, which is a 20% regression off of .net 8 for our workload. L3 max cache size is 48mb; however, .net 6 chose close to the l2 cache size (~40mb) off this specific hardware.

This said, the change should be a significant improvement of the current release., but would like to poke more at the end behavioral change.

[mrsharm]

Hi @jashook,

Now that you have confirmed that the previously assumed issue isn't applicable to your problem, we might need a non-intrusive GC Collect only trace to investigate this further by following the steps for Linux here using dotnet-trace.

One hypothesis is that the workload just survives more and therefore, your gen0 budget i.e., FinalYoungestDesired * Number of Heaps is much higher - we'll need to, of course, confirm this.