The Mac framework_tests_impeller LUCI shard is flaky.
Currently in dev/bots/suite_runners/run_framework_tests.dart, runImpeller executes the entire packages/flutter test suite in a single monolithic invocation:
Future<void> runImpeller() async {
printProgress('${green}Running packages/flutter tests $reset in Impeller$reset');
await runFlutterTest(
path.join(flutterRoot, 'packages', 'flutter'),
options: <String>['--enable-impeller'],
);
}
This runs a lot of stuff in one shard. 20,000 tests or so. If one test fails, retrying the shard re-runs all of these, which feels wasteful.
I looked at the last 500 try runs of Mac framework_tests_impeller. I noticed that failures spike during peak CI load windows (US midday and evening batch merges), whereas the post-merge builder (flutter/prod/Mac framework_tests_impeller) maintains a 100% pass rate over the last 100 runs.
Hourly Distribution of Try Failures (Pacific Time)
Hour (Pacific) | Total Builds | Failures | Failure Rate
-------------------------------------------------------
00:00 - 07:00 | 107 | 4 | 3.7% ####
08:00 - 11:00 | 155 | 2 | 1.3% ##
12:00 - 15:00 | 121 | 8 | 6.6% ######## (Peak US midday)
16:00 - 21:00 | 88 | 3 | 3.4% ###
22:00 - 23:00 | 27 | 4 | 14.8% #### (Late batch merges)
From those 500 try runs, failures were found in only four test files. I think these could use some attention, or be split out into their own shard or something.
Here's a report from AI on them:
| Failing Test Suite |
Failures |
Failure Mode |
packages/flutter/test/widgets/draggable_scrollable_sheet_test.dart |
10 |
Physics fling/drag animation timing |
packages/flutter/test/material/progress_indicator_test.dart |
4 |
Continuous spinning indicator timeout (did not complete [E]) |
packages/flutter/test/widgets/editable_text_cursor_test.dart |
2 |
Cursor blink animation timing |
packages/flutter/test/widgets/transform_test.dart |
2 |
3D perspective anti-aliasing mismatch |
Root Causes per Failing Suite
- Decoupled Host Clocks (
progress_indicator_test.dart): Progress indicators run infinite AnimationController.repeat() loops. Under runner CPU throttling, asynchronous microtasks and timer ticks fall behind real wall-clock time, causing tests to exceed timeouts (- did not complete [E]).
- Unbounded Fling Decay (
draggable_scrollable_sheet_test.dart): Ballistic fling simulations (tester.fling) followed by unbounded tester.pumpAndSettle() generate long micro-step settling tails that hit timeouts or fail precision thresholds (closeTo(.5, precisionErrorTolerance)) under thread starvation.
- Cursor Blink Thresholds (
editable_text_cursor_test.dart): 500ms blink timer periods oscillate close to pump thresholds under CPU throttling.
- Subpixel Anti-Aliasing (
transform_test.dart): Evaluates 80 distinct 3D perspective angles comparing composited layers to canvas drawing via strict byte-for-byte image matching (matchesReferenceImage). Metal Impeller's fractional rasterization produces 1-pixel color differences on edges.
Evidence & Recent Build Runs
- Attempt 1: Build 8669260472188289793 — Failed in
transform_test.dart: 3D transform renders the same with or without needsCompositing.
- Attempt 2: Build 8669260470147319649 — Failed in
transform_test.dart: 3D transform renders the same with or without needsCompositing.
- Attempt 3: Build 8669115127847759697 —
transform_test.dart passed; failed on 24 test timeouts in progress_indicator_test.dart.
- Attempt 4: Build 8669111987131570129 —
transform_test.dart passed; failed on 23 test timeouts in progress_indicator_test.dart.
The
Mac framework_tests_impellerLUCI shard is flaky.Currently in
dev/bots/suite_runners/run_framework_tests.dart,runImpellerexecutes the entirepackages/fluttertest suite in a single monolithic invocation:This runs a lot of stuff in one shard. 20,000 tests or so. If one test fails, retrying the shard re-runs all of these, which feels wasteful.
I looked at the last 500 try runs of
Mac framework_tests_impeller. I noticed that failures spike during peak CI load windows (US midday and evening batch merges), whereas the post-merge builder (flutter/prod/Mac framework_tests_impeller) maintains a 100% pass rate over the last 100 runs.Hourly Distribution of Try Failures (Pacific Time)
From those 500 try runs, failures were found in only four test files. I think these could use some attention, or be split out into their own shard or something.
Here's a report from AI on them:
packages/flutter/test/widgets/draggable_scrollable_sheet_test.dartpackages/flutter/test/material/progress_indicator_test.dartdid not complete [E])packages/flutter/test/widgets/editable_text_cursor_test.dartpackages/flutter/test/widgets/transform_test.dartRoot Causes per Failing Suite
progress_indicator_test.dart): Progress indicators run infiniteAnimationController.repeat()loops. Under runner CPU throttling, asynchronous microtasks and timer ticks fall behind real wall-clock time, causing tests to exceed timeouts (- did not complete [E]).draggable_scrollable_sheet_test.dart): Ballistic fling simulations (tester.fling) followed by unboundedtester.pumpAndSettle()generate long micro-step settling tails that hit timeouts or fail precision thresholds (closeTo(.5, precisionErrorTolerance)) under thread starvation.editable_text_cursor_test.dart): 500ms blink timer periods oscillate close to pump thresholds under CPU throttling.transform_test.dart): Evaluates 80 distinct 3D perspective angles comparing composited layers to canvas drawing via strict byte-for-byte image matching (matchesReferenceImage). Metal Impeller's fractional rasterization produces 1-pixel color differences on edges.Evidence & Recent Build Runs
transform_test.dart: 3D transform renders the same with or without needsCompositing.transform_test.dart: 3D transform renders the same with or without needsCompositing.transform_test.dartpassed; failed on 24 test timeouts inprogress_indicator_test.dart.transform_test.dartpassed; failed on 23 test timeouts inprogress_indicator_test.dart.