Cancelled vs failed runs

A cancelled CI run is not a failed run: cancellation means the pipeline was stopped before it reached a verdict, so it says nothing about whether the code was good or bad — and a failure rate that quietly folds cancellations in, with no way to see them separately, is measuring interruptions as well as quality.

How they differ

The distinction is in the run's conclusion. GitHub Actions' vocabulary is the clearest to reason with; GitLab exposes the same states under slightly different names:

The first three are decisive runs — they executed and produced a verdict. Cancelled and skipped runs did not, so a trustworthy failure rate uses decisive runs as its denominator and keeps the cancelled bucket countable on its own. Where tools legitimately differ is what to do with cancellations after that: the tidiest treatment drops them from both numerator and denominator, while TrimCI counts them on the unhealthy side, inside the decisive denominator, on the grounds that a pipeline which never delivered an answer is still CI that failed to deliver one — the reasoning is in decisive run. Either choice is defensible; hiding which one you made is not. One provider asymmetry worth knowing: GitHub Actions emits timed_out as its own conclusion, while GitLab CI reports a timed-out job as a plain failure, so the bucket only appears on one side.

Why cancellation is often good

A large share of cancellations are deliberate savings, not accidents. GitHub's concurrency groups with cancel-in-progress and GitLab's auto-cancellation of redundant pipelines both kill a run the moment a newer commit supersedes it. Every one of those cancelled runs stops burning compute early: you still pay for the minutes its jobs already executed — rounded up per job, and cancellation can itself take up to the 5-minute cancellation timeout — but the rest of the run is compute you never buy, and the queue slot is released early. On macOS runners those unspent minutes count roughly tenfold (see the macOS minute multiplier). A team with healthy auto-cancellation can show a large cancellation share while every run that actually reached a verdict passes.

The exception is humans mercy-killing runs: a developer cancelling a job that hung for 40 minutes is signal — usually about queue time or a hang — that deserves investigation even though it is still not a failure.

Why it matters

Blending the buckets invisibly corrupts metrics in both directions. Counting cancellations as failures inflates the failure rate — a repo with aggressive auto-cancellation can look like it fails half its runs while every run that reached a verdict passes — and tilts trend lines, since a busy week with many rapid pushes reads as a reliability regression. The reverse error is quieter: if failures can be cancelled mid-run and vanish from the denominator entirely, "cancel the red run" becomes an unintentional way to launder a bad change failure rate. Neither treatment is safe on its own, which is why keeping the three-way split — decisive, cancelled, skipped — visible matters more than the choice itself: a tool that folds cancellations into one side owes you both the separate count and an explicit caveat.

TrimCI computes every failure-rate denominator from decisive runs, consistently across dashboards and PDF report sections. Its judgement call is stated rather than hidden: a cancellation lands on the unhealthy side — inside the decisive denominator, counted as a failure — because a stuck or manually aborted pipeline is still unhealthy CI. The conclusion breakdown stays on screen as its own category, so the cancelled share is never invisible, and every report tells teams that auto-cancel superseded runs to read failure counts and failure cost as an upper bound.