Skip to content

Conversation

@jmagman
Copy link
Member

@jmagman jmagman commented Apr 2, 2022

Every test run shard first runs a 30 second "smoke test" suite to prove the test harness works. This suite runs 64 times per commit (once per shard/subshard/platform). Rename it to a "test harness tests" and only run it in the misc shard (which still runs 4 times per commit).

Running and passing in misc:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8818037355707764481/+/u/run_test.dart_for_framework_tests_shard_and_subshard_misc/test_stdout

Running test harness tests...
▌01:03:31▐ RUNNING: cd .; bin/cache/artifacts/engine/linux-x64/flutter_tester --help
▌01:03:31▐ ELAPSED TIME: 0.050s for bin/cache/artifacts/engine/linux-x64/flutter_tester --help in .
▌01:03:31▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pass_test.dart
▌01:03:44▐ ELAPSED TIME: 13.296s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pass_test.dart in dev/automated_tests
▌01:03:44▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/fail_test.dart
▌01:03:48▐ ELAPSED TIME: 3.625s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/fail_test.dart in dev/automated_tests
▌01:03:48▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pending_timer_fail_test.dart
▌01:03:52▐ ELAPSED TIME: 3.723s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/pending_timer_fail_test.dart in dev/automated_tests
▌01:03:52▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash1_test.dart
▌01:03:55▐ ELAPSED TIME: 3.491s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash1_test.dart in dev/automated_tests
▌01:03:55▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash2_test.dart
▌01:03:59▐ ELAPSED TIME: 3.357s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/crash2_test.dart in dev/automated_tests
▌01:03:59▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/syntax_error_test.broken_dart
▌01:04:01▐ ELAPSED TIME: 2.873s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/syntax_error_test.broken_dart in dev/automated_tests
▌01:04:01▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/missing_import_test.broken_dart
▌01:04:04▐ ELAPSED TIME: 2.831s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/missing_import_test.broken_dart in dev/automated_tests
▌01:04:04▐ RUNNING: cd dev/automated_tests; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/disallow_error_reporter_modification_test.dart
▌01:04:08▐ ELAPSED TIME: 3.885s for ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings test_smoke_test/disallow_error_reporter_modification_test.dart in dev/automated_tests
▌01:04:08▐ RUNNING: cd examples/api; ../../bin/flutter test --test-randomize-ordering-seed=20220401 --fatal-warnings --null-assertions --sound-null-safety

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@jmagman jmagman self-assigned this Apr 2, 2022
@flutter-dashboard flutter-dashboard bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Apr 2, 2022
@jmagman jmagman changed the title Only run test harness tests once instead of every shard Only run test harness tests in misc shard instead of every shard Apr 2, 2022
Copy link
Member

@zanderso zanderso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fluttergithubbot
Copy link
Contributor

This pull request is not suitable for automatic merging in its current state.

@jmagman jmagman merged commit bc0ec85 into flutter:master Apr 5, 2022
@jmagman jmagman deleted the smoketests branch April 5, 2022 00:27
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Apr 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants