diff --git a/testing/scenario_app/README.md b/testing/scenario_app/README.md index ce80b24246356..470eb320fb116 100644 --- a/testing/scenario_app/README.md +++ b/testing/scenario_app/README.md @@ -1,85 +1,43 @@ # Scenario App -This folder contains e2e integration tests for the engine in conjunction with a -fake dart:ui framework running in JIT or AOT. +[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22) -It intentionally has no dependencies on the Flutter framework or tooling, such -that it should be buildable as a presubmit or postsubmit to the engine even in -the face of changes to Dart or dart:ui that require upstream changes in the -Flutter tooling. +This package simulates a Flutter app that uses the engine (`dart:ui`) only, +in conjunction with Android and iOS-specific embedding code that simulates the +use of the engine in a real app (such as plugins and platform views). -## Adding a New Scenario +The [`run_android_tests.sh`](run_android_tests.sh) and +[`run_ios_tests.sh`](run_ios_tests.sh) are then used to run the tests on a +connected device or emulator. -Create a new subclass of [Scenario](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/scenario.dart#L9) -and add it to the map in [scenarios.dart](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/lib/src/scenarios.dart#L22). -For an example, see [animated_color_square.dart](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/animated_color_square.dart#L15), -which draws a continuously animating colored square that bounces off the sides -of the viewport. +See also: -Then set the scenario from the Android or iOS app by calling "set_scenario" on -platform channel. +- [`bin/`](bin/), the entry point for running Android integration tests. +- [`lib/`](lib/), the Dart code and instrumentation for the scenario app. +- [`ios/`](ios/), the iOS-side native code and tests. +- [`android/`](android/), the Android-side native code and tests. -## Running for iOS +## Running a smoke test on Firebase TestLab -Build the `ios_debug_sim_unopt` engine variant, and run +To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, +and run [`./ci/firebase_testlab.py`](../../ci/firebase_testlab.py), or pass +`--variant` to run a different configuration. ```sh -./run_ios_tests.sh -``` - -in your shell. - -To run or debug in Xcode, open the xcodeproj file located in -`/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`. - -### iOS Platform View Tests - -For PlatformView tests on iOS, you'll also have to edit the dictionaries in -[AppDelegate.m](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/Scenarios/AppDelegate.m#L29) and [GoldenTestManager.m](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenTestManager.m#L25) so that the correct golden image can be found. Also, you'll have to add a [GoldenPlatformViewTests](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenPlatformViewTests.h#L18) in [PlatformViewUITests.m](https://github.com/flutter/engine/blob/af2ffc02b72af2a89242ca3c89e18269b1584ce5/testing/scenario_app/ios/Scenarios/ScenariosUITests/PlatformViewUITests.m). - -If `PlatformViewRotation` is failing, make sure Simulator app Device > Rotate Device Automatically -is selected, or run: - -```bash -defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1 +# From the root of the engine repository +$ ./ci/firebase_testlab.py --variant android_debug_arm64 ``` -### Generating Golden Images on iOS - -Screenshots are saved as -[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s. -If you look at the output from running the tests you'll find a path in the form: -`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH`. -Inside that directory you'll find -`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` which is where all the images that were -compared against golden reside. - -## Running for Android +> [!NOTE] +> These instructions were not verified at the time of writing/refactoring. -### Integration tests - -For emulators running on a x64 host, build `android_debug_unopt_x64` using -`./tools/gn --android --unoptimized --goma --android-cpu=x64`. - -Then, launch the emulator, and run `./testing/scenario_app/run_android_tests.sh android_debug_unopt_x64`. - -If you wish to build a different engine variant, make sure to pass that variant to the script `run_android_tests.sh`. - -If you make a change to the source code, you would need to rebuild the same engine variant. - -### Smoke test on FTL - -To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, and run -`./flutter/ci/firebase_testlab.py`. If you wish to test a different variant, e.g. -debug arm64, pass `--variant android_debug_arm64`. - -### Updating Gradle dependencies - -If a Gradle dependency is updated, lockfiles must be regenerated. +## Adding a New Scenario -To generate new lockfiles, run: +Create a new subclass of [Scenario](lib/src/scenario.dart) and add it to the map +in [scenarios.dart](lib/src/scenarios.dart). For an example, see +[animated_color_square.dart](lib/src/animated_color_square.dart), which draws a +continuously animating colored square that bounces off the sides of the +viewport. -```bash -cd android/app -../../../../../third_party/gradle/bin/gradle generateLockfiles -``` +Then set the scenario from the Android or iOS app by calling `set_scenario` on +platform channel `driver`. diff --git a/testing/scenario_app/android/README.md b/testing/scenario_app/android/README.md new file mode 100644 index 0000000000000..ec4f1857bc665 --- /dev/null +++ b/testing/scenario_app/android/README.md @@ -0,0 +1,42 @@ +# Scenario App: Android Tests + +As mentioned in the [top-level README](../README.md), this directory contains +the Android-specific native code and tests for the [scenario app](../lib). To +run the tests, you will need to build the engine with the appropriate +configuration. + +For example, `android_debug_unopt` or `android_debug_unopt_arm64` was built, +run: + +```sh +# From the root of the engine repository +$ ./testing/run_android_tests.sh android_debug_unopt + +# Or, for arm64 +$ ./testing/run_android_tests.sh android_debug_unopt_arm64 +``` + +## CI Configuration + +See [`ci/builders/linux_android_emulator.json`](../../../ci/builders/linux_android_emulator.json) +, and grep for `run_android_tests.sh`. + +The following matrix of configurations is tested on the CI: + +| API Version | Graphics Backend | Skia Gold | Rationale | +| ----------- | ------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------- | +| 28 | Skia | [Android 28 + Skia][skia-gold-skia-28] | Older Android devices (without `ImageReader`) on Skia. | +| 28 | Impeller (OpenGLES) | [Android 28 + Impeller OpenGLES][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. | +| 34 | Skia | [Android 34 + Skia][skia-gold-skia-34] | Newer Android devices on Skia. | +| 34 | Impeller (OpenGLES) | [Android 34 + Impeller OpenGLES][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES. | +| 34 | Impeller (Vulkan) | [Android 34 + Impeller Vulkan][skia-gold-impeller-vulkan-34] | Newer Android devices on Impeller. | + +[skia-gold-skia-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dskia&negative=true&positive=true +[skia-gold-impeller-opengles-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true +[skia-gold-skia-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia&negative=true&positive=true +[skia-gold-impeller-opengles-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true +[skia-gold-impeller-vulkan-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true + +## Updating Gradle dependencies + +See [Updating the Embedding Dependencies](../../../tools/cipd/android_embedding_bundle/README.md). diff --git a/testing/scenario_app/bin/README.md b/testing/scenario_app/bin/README.md new file mode 100644 index 0000000000000..7964d6e5960cc --- /dev/null +++ b/testing/scenario_app/bin/README.md @@ -0,0 +1,40 @@ +# `android_integration_tests` runner + +This directory contains code specific to running Android integration tests. + +The tests are uploaded and run on the device using `adb`, and screenshots are +captured and compared using Skia Gold (if available, for example on CI). + +## Usage + +```sh +dart bin/android_integration_tests.dart \ + --adb ../third_party/android_tools/sdk/platform-tools/adb \ + --out-dir ../out/android_debug_unopt_arm64 +``` + +## Debugging + +When debugging, you can use the `--smoke-test` argument to run a single test +by class name, which can be useful to verify the setup. + +For example, to run the `EngineLaunchE2ETest` test: + +```sh +dart bin/android_integration_tests.dart \ + --adb ../third_party/android_tools/sdk/platform-tools/adb \ + --out-dir ../out/android_debug_unopt_arm64 \ + --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest +``` + +## Additional arguments + +- `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true + when running on CI, and false otherwise (i.e. when running locally). If + set to true, [isSkiaGoldClientAvailable] must be true. + +- `--enable-impeller`: Enable Impeller for the Android app. Defaults to + false, which means that the app will use Skia as the graphics backend. + +- `--impeller-backend`: The Impeller backend to use for the Android app. + Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true. diff --git a/testing/scenario_app/bin/android_integration_tests.dart b/testing/scenario_app/bin/android_integration_tests.dart index 3332de5f7f81d..98b8b7abcb39c 100644 --- a/testing/scenario_app/bin/android_integration_tests.dart +++ b/testing/scenario_app/bin/android_integration_tests.dart @@ -16,8 +16,7 @@ import 'utils/logs.dart'; import 'utils/process_manager_extension.dart'; import 'utils/screenshot_transformer.dart'; -const int tcpPort = 3001; - +// If you update the arguments, update the documentation in the README.md file. void main(List args) async { final ArgParser parser = ArgParser() ..addOption( @@ -82,6 +81,8 @@ void main(List args) async { ); } +const int _tcpPort = 3001; + enum _ImpellerBackend { vulkan, opengles; @@ -137,7 +138,7 @@ Future _run({ late ServerSocket server; final List> pendingComparisons = >[]; await step('Starting server...', () async { - server = await ServerSocket.bind(InternetAddress.anyIPv4, tcpPort); + server = await ServerSocket.bind(InternetAddress.anyIPv4, _tcpPort); stdout.writeln('listening on host ${server.address.address}:${server.port}'); server.listen((Socket client) { stdout.writeln('client connected ${client.remoteAddress.address}:${client.remotePort}'); @@ -237,7 +238,7 @@ Future _run({ }); await step('Reverse port...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'reverse', 'tcp:3000', 'tcp:$tcpPort']); + final int exitCode = await pm.runAndForward([adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']); if (exitCode != 0) { panic(['could not forward port']); } diff --git a/testing/scenario_app/ios/README.md b/testing/scenario_app/ios/README.md new file mode 100644 index 0000000000000..972625056d403 --- /dev/null +++ b/testing/scenario_app/ios/README.md @@ -0,0 +1,55 @@ +# Scenario App: iOS Tests + +As mentioned in the [top-level README](../README.md), this directory contains +the iOS-specific native code and tests for the [scenario app](../lib). To run +the tests, you will need to build the engine with the appropriate configuration. + +For example, after building `ios_debug_sim_unopt` (to run on Intel Macs) or `ios_debug_sim_unopt_arm64` (to run on ARM Macs), +run: + +```sh +# From the root of the engine repository +$ ./testing/run_ios_tests.sh ios_debug_sim_unopt +``` +or +```sh +# From the root of the engine repository +$ ./testing/run_ios_tests.sh ios_debug_sim_unopt_arm64 +``` + +To run or debug in Xcode, open the xcodeproj file located in +`/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`. + +## CI Configuration + +See [`ci/builders/mac_unopt.json`](../../../../ci/builders/mac_unopt.json), and +grep for `run_ios_tests.sh`. + +## iOS Platform View Tests + +For PlatformView tests on iOS, edit the dictionaries in +[AppDelegate.m](Scenarios/Scenarios/AppDelegate.m) and +[GoldenTestManager.m](Scenarios/ScenariosUITests/GoldenTestManager.m) so that +the correct golden image can be found. Also, add a +[GoldenPlatformViewTests](Scenarios/ScenariosUITests/GoldenPlatformViewTests.h) +in [PlatformViewUITests.m](Scenarios/ScenariosUITests/PlatformViewUITests.m). + +If `PlatformViewRotation` is failing, make sure +`Simulator app Device > Rotate Device Automatically` is selected, or run: + +```bash +defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1 +``` + +## Generating Golden Images on iOS + +Screenshots are saved as +[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s. + +A path in the form of +`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH` will be +printed to the console. + +Inside that directory there is a directory +`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` +which is where all the images that were compared against golden reside.