[Android][Testing] Run tests on all supported Android API levels

[simonrozsival]

Part of #107868, follow-up to #117716

This PR won't be merged, it's for testing purposes only.

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @vitek-karas, @simonrozsival, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[simonrozsival]

/azp run runtime-androidemulator

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[simonrozsival]

Tests did not show any blockers for the .NET 10 release. Some notes:

Across API levels:

• MemoryStream_CapacityBoundaryChecks fails with OutOfMemoryException: MemoryStream_CapacityBoundaryChecks fails with OutOfMemoryException #119839
• Ping tests failing: [Android][arm64] System.Net.NetworkInformation.Tests.UnixPingUtilityTests.TimeoutIsRespected fails with out of range values #119852

API 36

• There are several failures caused by XHarness exit code: 84 (DEVICE_FILE_COPY_FAILURE)
  • It appears all the tests are passing though

Older API levels

API 23

• the test processes seem to crash randomly with signal 11:

  09-25 14:54:46.723  2837  4130 I DOTNET  : 	[PASS] System.Net.Sockets.Tests.DualModeConnectionlessReceiveMessageFrom.BeginReceiveMessageFromV4BoundToAnyMappedV4_Success
  09-25 14:54:46.724  2837  4130 I DOTNET  : 	[PASS] System.Net.Sockets.Tests.DualModeConnectionlessReceiveMessageFrom.BeginReceiveMessageFromV4BoundToSpecificV4_Success
  09-25 14:54:46.827  2837  4130 I DOTNET  : 	[PASS] System.Net.Sockets.Tests.DualModeConnectionlessReceiveMessageFrom.ReceiveMessageFromV4BoundToAnyMappedV4_Success
  09-25 14:54:46.829  2837  4130 I DOTNET  : 	[PASS] System.Net.Sockets.Tests.DualModeConnectionlessReceiveMessageFrom.BeginReceiveMessageFromV4BoundToAnyV6_Success
  09-25 14:54:46.858  1251  1251 I Zygote  : Process 2837 exited due to signal (11)
  09-25 14:54:46.862  1546  2312 I ActivityManager: Process net.dot.System.Net.Sockets.Tests (pid 2837) has died
  09-25 14:54:46.863  1546  2312 W ActivityManager: Crash of app net.dot.System.Net.Sockets.Tests running instrumentation ComponentInfo{net.dot.System.Net.Sockets.Tests/net.dot.MonoRunner}
  09-25 14:54:46.866  2829  2829 D AndroidRuntime: Shutting down VM
  09-25 14:54:47.089  1546  2312 I ActivityManager: Force stopping net.dot.System.Net.Sockets.Tests appid=10056 user=0: finished inst
  09-25 14:54:47.101  1546  1569 W ActivityManager: Error shutting down UiAutomationConnection
  

API 21 + 22

• several types of failures on the Ubuntu.2204.Amd64.Android.21.Open queue:
  • CoreCLR initialization fails

    Console log: 'Microsoft.Extensions.Options.Tests' from job b321fc11-7fd0-413e-b276-2048ccbc7c4d workitem a6f1f09e-3f1e-40da-89a2-80d02b9c152a (ubuntu.2204.amd64.android.21.open.rt) executed on machine a0004AW running Linux-6.8.0-1031-azure-x86_64-with-glibc2.35
    ...
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/System.Reflection.Primitives.dll
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/xunit-excludes.txt
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/System.Net.ServicePoint.dll
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/System.Net.NameResolution.dll
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/AndroidTestRunner.dll
    I/DOTNET  ( 2526): Extracting asset to /data/data/net.dot.Microsoft.Extensions.Options.Tests/files/Newtonsoft.Json.dll
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_setEnv:
    D/DOTNET  ( 2526): Setting env: HOME=/data/data/net.dot.Microsoft.Extensions.Options.Tests/files
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_setEnv:
    D/DOTNET  ( 2526): Setting env: ASSETS_DIR=/data/data/net.dot.Microsoft.Extensions.Options.Tests/files
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_setEnv:
    D/DOTNET  ( 2526): Setting env: TMPDIR=/data/data/net.dot.Microsoft.Extensions.Options.Tests/cache
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_setEnv:
    D/DOTNET  ( 2526): Setting env: TEST_RESULTS_DIR=/data/data/net.dot.Microsoft.Extensions.Options.Tests/cache
    I/DOTNET  ( 2526): MonoRunner initializeRuntime, entryPointLibName=AndroidTestRunner.dll
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_initRuntime (CoreCLR):
    D/DOTNET  ( 2526): mono_droid_runtime_init (CoreCLR) called with executable: AndroidTestRunner.dll
    D/DOTNET  ( 2526): Calling coreclr_initialize
    D/DOTNET  ( 2526): coreclr_initialize returned 0x8000ffff
    E/DOTNET  ( 2526): Failed to initialize runtime, return-code=-2147418113
    D/DOTNET  ( 2526): Java_net_dot_MonoRunner_freeNativeResources (CoreCLR):
    I/art     ( 2526): System.exit called, status: -2147418113
    I/AndroidRuntime( 2526): VM exiting with result code -2147418113, cleanup skipped.
    I/Zygote  ( 1127): Process 2526 exited cleanly (255)
    I/ActivityManager( 1472): Process net.dot.Microsoft.Extensions.Options.Tests (pid 2526) has died
    W/ActivityManager( 1472): Crash of app net.dot.Microsoft.Extensions.Options.Tests running instrumentation ComponentInfo{net.dot.Microsoft.Extensions.Options.Tests/net.dot.MonoRunner}
    D/AndroidRuntime( 2515): Shutting down VM
    I/ActivityManager( 1472): Force stopping net.dot.Microsoft.Extensions.Options.Tests appid=10053 user=0: finished inst
    

  • library name too long:

    Console log: 'Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests' from job b321fc11-7fd0-413e-b276-2048ccbc7c4d workitem 9c28a054-49c4-4621-948d-63fb3b7bf250 (ubuntu.2204.amd64.android.21.open.rt) executed on machine a0004B2 running Linux-6.8.0-1031-azure-x86_64-with-glibc2.35
    ...
    --------- beginning of main
    W/art     ( 2498): No such thread id for suspend: 9
    D/AndroidRuntime( 2513): 
    D/AndroidRuntime( 2513): >>>>>> AndroidRuntime START com.android.internal.os.RuntimeInit <<<<<<
    D/AndroidRuntime( 2513): CheckJNI is ON
    E/memtrack( 2513): Couldn't load memtrack module (No such file or directory)
    E/android.os.Debug( 2513): failed to load memtrack module: -2
    D/AndroidRuntime( 2513): Calling main entry com.android.commands.am.Am
    --------- beginning of system
    I/ActivityManager( 1471): Force stopping net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests appid=10053 user=0: start instr
    I/ActivityManager( 1471): Start proc net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests for added application net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests: pid=2524 uid=10053 gids={50053, 9997, 3003, 1028, 1015} abi=x86_64
    I/art     ( 2524): Not late-enabling -Xcheck:jni (already on)
    E/art     ( 2524): dlopen("/data/app/net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests-1/lib/x86_64/libSystem.Security.Cryptography.Native.Android.so", RTLD_LAZY) failed: dlopen failed: library name "/data/app/net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests-1/lib/x86_64/libSystem.Security.Cryptography.Native.Android.so" too long
    D/AndroidRuntime( 2524): Shutting down VM
    --------- beginning of crash
    E/AndroidRuntime( 2524): FATAL EXCEPTION: main
    E/AndroidRuntime( 2524): Process: net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests, PID: 2524
    E/AndroidRuntime( 2524): java.lang.UnsatisfiedLinkError: dlopen failed: library name "/data/app/net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests-1/lib/x86_64/libSystem.Security.Cryptography.Native.Android.so" too long
    E/AndroidRuntime( 2524): 	at java.lang.Runtime.loadLibrary(Runtime.java:371)
    E/AndroidRuntime( 2524): 	at java.lang.System.loadLibrary(System.java:989)
    E/AndroidRuntime( 2524): 	at net.dot.MonoRunner.<clinit>(MonoRunner.java:37)
    E/AndroidRuntime( 2524): 	at java.lang.reflect.Constructor.newInstance(Native Method)
    E/AndroidRuntime( 2524): 	at java.lang.Class.newInstance(Class.java:1572)
    E/AndroidRuntime( 2524): 	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4456)
    E/AndroidRuntime( 2524): 	at android.app.ActivityThread.access$1500(ActivityThread.java:144)
    E/AndroidRuntime( 2524): 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1339)
    E/AndroidRuntime( 2524): 	at android.os.Handler.dispatchMessage(Handler.java:102)
    E/AndroidRuntime( 2524): 	at android.os.Looper.loop(Looper.java:135)
    E/AndroidRuntime( 2524): 	at android.app.ActivityThread.main(ActivityThread.java:5221)
    E/AndroidRuntime( 2524): 	at java.lang.reflect.Method.invoke(Native Method)
    E/AndroidRuntime( 2524): 	at java.lang.reflect.Method.invoke(Method.java:372)
    E/AndroidRuntime( 2524): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:899)
    E/AndroidRuntime( 2524): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:694)
    W/ActivityManager( 1471): Error in app net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests running instrumentation ComponentInfo{net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests/net.dot.MonoRunner}:
    W/ActivityManager( 1471):   java.lang.UnsatisfiedLinkError
    W/ActivityManager( 1471):   java.lang.UnsatisfiedLinkError: dlopen failed: library name "/data/app/net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests-1/lib/x86_64/libSystem.Security.Cryptography.Native.Android.so" too long
    I/ActivityManager( 1471): Force stopping net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests appid=10053 user=0: finished inst
    I/ActivityManager( 1471): Killing 2524:net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests/u0a53 (adj 0): stop net.dot.Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests
    I/art     ( 1127): Explicit concurrent mark sweep GC freed 708(30KB) AllocSpace objects, 0(0B) LOS objects, 90% free, 110KB/1134KB, paused 466us total 26.555ms
    D/AndroidRuntime( 2513): Shutting down VM
    I/art     ( 1127): Explicit concurrent mark sweep GC freed 5(160B) AllocSpace objects, 0(0B) LOS objects, 90% free, 110KB/1134KB, paused 425us total 18.762ms
    I/art     ( 1127): Explicit concurrent mark sweep GC freed 5(160B) AllocSpace objects, 0(0B) LOS objects, 90% free, 110KB/1134KB, paused 423us total 4.383ms
    D/AlarmManagerService( 1471): Kernel timezone updated to 0 minutes west of GMT
    D/AlarmManagerService( 1471): Setting time of day to sec=1758810501
    V/AlarmClock( 2102): AlarmInitReceiver android.intent.action.TIME_SET
    V/AlarmClock( 2102): AlarmInitReceiver finished
    

  • no compatible device found:

    Console log: 'IcuAppLocal.Tests' from job b321fc11-7fd0-413e-b276-2048ccbc7c4d workitem dbd5a468-9e95-461a-9b56-80595f3c574b (ubuntu.2204.amd64.android.21.open.rt) executed on machine a0004AV running Linux-6.8.0-1031-azure-x86_64-with-glibc2.35
    + export __TestArchitecture=x64
    ...
    [10.0.0-prerelease.25427.1+dbb478b6aafa222529ae95be344a1b91485c4adf] XHarness command issued: android test --app IcuAppLocal.Tests.apk --output-directory /datadisks/disk1/work/ABC10992/w/AE85092E/uploads --timeout 00:30:00 --package-name net.dot.IcuAppLocal.Tests -v --instrumentation net.dot.MonoRunner
    [14:21:14] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/ABC10992/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25427.1/tools/net8.0/any/../../../runtimes/any/native/adb/linux/adb
    [14:21:15] dbug: Full resolved path:'/datadisks/disk1/work/ABC10992/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25427.1/runtimes/any/native/adb/linux/adb'
    [14:21:15] info: Will attempt to find device supporting architectures: 'x86_64'
    [14:21:15] dbug: Executing command: '/datadisks/disk1/work/ABC10992/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25427.1/runtimes/any/native/adb/linux/adb start-server'
    [14:21:15] dbug: 
    [14:21:15] info: Finding attached devices/emulators...
    [14:21:15] dbug: Executing command: '/datadisks/disk1/work/ABC10992/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25427.1/runtimes/any/native/adb/linux/adb devices -l'
    [14:21:15] dbug: Found 1 possible devices
    [14:21:15] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
    [14:21:15] dbug: Executing command: '/datadisks/disk1/work/ABC10992/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25427.1/runtimes/any/native/adb/linux/adb -s emulator-5554 shell getprop ro.product.cpu.abilist'
    [14:21:15] fail: No attached device supports one of required architectures x86_64
    [14:21:15] dbug: No suitable devices found
    [14:21:15] crit: Failed to find compatible device: x86_64
    [14:21:15] dbug: Saving diagnostics data to '/datadisks/disk1/work/ABC10992/w/AE85092E/e/diagnostics.json'
    XHarness exit code: 81 (DEVICE_NOT_FOUND)