Skip to content

Mac framework_tests_impeller is too big and flaky #193689

Description

@andywolff

The Mac framework_tests_impeller LUCI shard is flaky.

Currently in dev/bots/suite_runners/run_framework_tests.dart, runImpeller executes the entire packages/flutter test suite in a single monolithic invocation:

Future<void> runImpeller() async {
  printProgress('${green}Running packages/flutter tests $reset in Impeller$reset');
  await runFlutterTest(
    path.join(flutterRoot, 'packages', 'flutter'),
    options: <String>['--enable-impeller'],
  );
}

This runs a lot of stuff in one shard. 20,000 tests or so. If one test fails, retrying the shard re-runs all of these, which feels wasteful.

I looked at the last 500 try runs of Mac framework_tests_impeller. I noticed that failures spike during peak CI load windows (US midday and evening batch merges), whereas the post-merge builder (flutter/prod/Mac framework_tests_impeller) maintains a 100% pass rate over the last 100 runs.

Hourly Distribution of Try Failures (Pacific Time)

Hour (Pacific)   | Total Builds | Failures | Failure Rate
-------------------------------------------------------
00:00 - 07:00    |          107 |        4 |         3.7%  ####
08:00 - 11:00    |          155 |        2 |         1.3%  ##
12:00 - 15:00    |          121 |        8 |         6.6%  ########  (Peak US midday)
16:00 - 21:00    |           88 |        3 |         3.4%  ###
22:00 - 23:00    |           27 |        4 |        14.8%  ####      (Late batch merges)

From those 500 try runs, failures were found in only four test files. I think these could use some attention, or be split out into their own shard or something.

Here's a report from AI on them:

Failing Test Suite Failures Failure Mode
packages/flutter/test/widgets/draggable_scrollable_sheet_test.dart 10 Physics fling/drag animation timing
packages/flutter/test/material/progress_indicator_test.dart 4 Continuous spinning indicator timeout (did not complete [E])
packages/flutter/test/widgets/editable_text_cursor_test.dart 2 Cursor blink animation timing
packages/flutter/test/widgets/transform_test.dart 2 3D perspective anti-aliasing mismatch

Root Causes per Failing Suite

  1. Decoupled Host Clocks (progress_indicator_test.dart): Progress indicators run infinite AnimationController.repeat() loops. Under runner CPU throttling, asynchronous microtasks and timer ticks fall behind real wall-clock time, causing tests to exceed timeouts (- did not complete [E]).
  2. Unbounded Fling Decay (draggable_scrollable_sheet_test.dart): Ballistic fling simulations (tester.fling) followed by unbounded tester.pumpAndSettle() generate long micro-step settling tails that hit timeouts or fail precision thresholds (closeTo(.5, precisionErrorTolerance)) under thread starvation.
  3. Cursor Blink Thresholds (editable_text_cursor_test.dart): 500ms blink timer periods oscillate close to pump thresholds under CPU throttling.
  4. Subpixel Anti-Aliasing (transform_test.dart): Evaluates 80 distinct 3D perspective angles comparing composited layers to canvas drawing via strict byte-for-byte image matching (matchesReferenceImage). Metal Impeller's fractional rasterization produces 1-pixel color differences on edges.

Evidence & Recent Build Runs

  • Attempt 1: Build 8669260472188289793 — Failed in transform_test.dart: 3D transform renders the same with or without needsCompositing.
  • Attempt 2: Build 8669260470147319649 — Failed in transform_test.dart: 3D transform renders the same with or without needsCompositing.
  • Attempt 3: Build 8669115127847759697 — transform_test.dart passed; failed on 24 test timeouts in progress_indicator_test.dart.
  • Attempt 4: Build 8669111987131570129 — transform_test.dart passed; failed on 23 test timeouts in progress_indicator_test.dart.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    c: flakeTests that sometimes, but not always, incorrectly passteam-infraOwned by Infrastructure team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions