Flaky test

A flaky test is a test that both passes and fails on the same code — nothing changed in the source, the test, or its inputs, yet the outcome differs between runs. Flakiness makes a red build ambiguous: it might mean "this change is broken," or it might mean nothing at all.

Common causes

Most flakiness traces back to nondeterminism the test did not control:

• Timing — fixed sleeps, race conditions, async operations awaited too loosely.
• Shared state — test-order dependence, leaked fixtures, reused databases or caches.
• External dependencies — real network calls, third-party APIs, DNS, clock skew.
• Infrastructure — resource contention on busy runners, container cold starts, low memory.

How it is detected

There are two levels of detection, with very different data requirements.

Per-test detection operates on test-level results. It needs a test report — JUnit XML or in-build instrumentation — uploaded from every pipeline, and it flags an individual test that reports different conclusions for the same commit. Tools such as Trunk Flaky Tests and BuildPulse work this way; the trade-off is adding an uploader or agent to every build.

Pipeline-level inference operates on run metadata alone. Fail-then-pass alternation on the same branch, and retried runs that go green on a later attempt without a new commit, are strong flakiness signals visible from provider APIs without touching the build. TrimCI works at this level: it infers flaky behavior from run alternation and retry metadata, and it does not do per-test detection — it never ingests test reports or reads code, so it cannot name the individual test. What it names are the repositories and pipelines that behave flakily, and what that behavior costs.

Why it matters

Flakiness is common even in mature suites — almost 16% of Google's 4.2 million tests showed some level of flakiness (J. Micco, The State of Continuous Integration Testing @Google, ICST 2017), still the most-cited public figure of its kind as of 2026-07-28. The damage is mostly indirect: once a team assumes a red build might be flaky, real failures get re-run instead of investigated, and every retry adds to the retry tax — compute paid twice for zero new information. Flaky failures also distort failure-rate metrics unless the denominator is handled carefully (see decisive runs).

TrimCI surfaces flaky behavior in its PDF reports at the pipeline-and-branch level — how many times a pipeline flipped between pass and fail across its last decisive runs — plus, for GitHub Actions, the share of runs that were retried and an approximate count of the minutes those extra attempts burned — as part of broader CI pipeline failure analysis, so the fix list starts with the pipelines whose flakiness costs the most. The two signals are reported separately on purpose: a same-commit retry overwrites the earlier attempt when the run is synced, so it never shows up as an alternation in the flip table.