Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Conver xharness, address feedback
  • Loading branch information
EgorBo committed May 12, 2020
commit d65f54f9af6578050a4ca9e456464dc9a129b1d4
19 changes: 11 additions & 8 deletions docs/workflow/testing/libraries/testing-android.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ To build the tests and run them on Android (devices or emulators) you need the f

(TODO: provide command-line steps to download them with dependencies)

Once SDKs are downloaded, set `ANDROID_NDK_ROOT`(CLARIFY: looks like `ANDROID_NDK_HOME` should also be set) and `ANDROID_SDK_ROOT`.
Once SDKs are downloaded, set `ANDROID_NDK_ROOT` and `ANDROID_SDK_ROOT`.
Example:
```
export ANDROID_SDK_ROOT=/Users/egorbo/Library/Android/sdk
export ANDROID_NDK_ROOT=/Users/egorbo/android-ndk-r21b
export ANDROID_NDK_HOME=$ANDROID_NDK_ROOT
```

Next, we need OpenSSL binaries and headers, we haven't properly integrated this dependency yet (it's an ongoing discussion) but there is a workaround:
Expand All @@ -26,7 +25,7 @@ export AndroidOpenSslHeaders="$GOOGLE_OPENSSL/ssl/include"
export AndroidOpenSslCryptoLib="$GOOGLE_OPENSSL/crypto/libs/android.x86_64/libcrypto.so"
export AndroidOpenSslLib="$GOOGLE_OPENSSL/ssl/libs/android.x86_64/libssl.so"
```
**IMPORTANT:** make sure correct ABIs are used in the path, e.g. `-arch x64` -> `android.x86_64` (TODO: auto-detect)
**IMPORTANT:** make sure correct ABIs are used in the path, e.g. `-arch x64` -> `android.x86_64`

Now we're ready to build everything for Android:
```
Expand All @@ -36,20 +35,24 @@ and even run tests one by one for each test suite:
```
./build.sh libs.tests -os Android -arch x64 -test
```
Make sure an emulator is booted or a device is plugged and unlocked.
Make sure an emulator is booted (see `AVD Manager`) or a device is plugged in and unlocked.

**NOTE**: Xharness doesn't run any UI on Android and runs tests using headless testing API so the device/emulator won't show anything (but still must stay active).

### Running individual test suites
- The following shows how to run tests for a specific library
```
cd src/libraries/System.Numerics.Vectors/tests
~runtime/dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
```

### How the tests work
Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/master/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
runner called XHarness TestRunner which runs tests for all *.Tests.dll libs in the bundle. There is also XHarness.CLI tool with ADB inside to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed.

### Obtaining the logs
XHarness doesn't talk much and only saves tests result at the end to a file but you can also subscribe to live logs via the following command:
XHarness for Android doesn't talk much and only saves tests result to a file once tests finished but you can also subscribe to live logs via the following command:
```
adb logcat -S DOTNET
adb logcat -s "DOTNET"
```
Or simply open `logcat` window in Android Studio or Visual Stuido.

Expand Down
15 changes: 9 additions & 6 deletions docs/workflow/testing/libraries/testing-apple.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Testing Libraries on iOS and tvOS

In order to build librarires and tests for iOS or tvOS you only need some fresh version of XCode installed (e.g. 11.3 or higher).
In order to build libraries and tests for iOS or tvOS you only need some fresh version of XCode installed (e.g. 11.3 or higher).

Build Librarires for iOS:
Build Libraries for iOS:
```
./build.sh mono+libs -os iOS -arch x64
```
Expand All @@ -16,14 +16,17 @@ Unlike XHarness-Android, XHarness for iOS is able to boot simulators by its own.
- The following shows how to run tests for a specific library on a simulator
```
cd src/libraries/System.Numerics.Vectors/tests
~runtime/dotnet.sh build /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
./dotnet.sh build /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
```
for devices you need to specify `DevTeamProvisioning`:
```
cd src/libraries/System.Numerics.Vectors/tests
~runtime/dotnet.sh build /t:Test /p:TargetOS=iOS /p:TargetArchitecture=arm64 /p:DevTeamProvisioning=...
./dotnet.sh build /t:Test /p:TargetOS=iOS /p:TargetArchitecture=arm64 /p:DevTeamProvisioning=...
```
AppleAppBuilder generates temp Xcode projects you can manually open and resolve provisioning there using native UI and deploy to your devices and start debugging.

### How the tests work
iOS/tvOS *.app (or *.ipa) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/master/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
runner called XHarness TestRunner which runs tests for all *.Tests.dll libs in the bundle. There is also XHarness.CLI tool with `mlaunch` inside to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via sockets.

### Known Issues
- Most of the test suites crash on devices due to [#35674)(https://github.com/dotnet/runtime/issues/35674)
- Most of the test suites crash on devices due to [#35674)(https://github.com/dotnet/runtime/issues/35674)