MTTR in CI/CD

MTTR in CI/CD (mean time to recovery) is the time a pipeline spends broken, measured from the first failing run to the next passing run of the same pipeline on the same branch. It answers a narrow, practical question: when CI goes red, how long until it is green again?

How it is measured

A recovery interval opens when a pipeline that was passing produces a failing run, and closes with the next successful run of the same pipeline on the same branch. Consecutive failures inside the interval do not open new intervals — they are the same breakage continuing. MTTR over a time window aggregates those red-to-green intervals. It uses a deliberately narrower population than the decisive runs behind failure rates: cancellations are dropped entirely rather than counted as failures, because an aborted run neither breaks a pipeline nor repairs one — counting them would fabricate red streaks nobody ever saw.

Median versus mean matters more here than for most metrics. Recovery times are heavily right-skewed — most breakages are fixed in minutes, a few sit unfixed over a weekend. One Friday failure repaired on Monday can triple the mean while the median barely moves. The median describes the typical recovery; the mean captures the cost of the outliers. A useful report shows both, because a low median with a high mean means the process works except when it completely does not.

Pipeline vs incident MTTR

Most glossaries conflate two different metrics. Incident-ops MTTR — time to restore service after a production incident, tracked among the DORA metrics, where it was long called "time to restore service" and is now named failed deployment recovery time (DORA's software delivery performance metrics guide, as of 2026-07-28) — starts at user-facing impact and ends at service recovery. Pipeline MTTR starts at a failing CI run and ends at a passing one; production may be entirely unaffected the whole time. The two correlate loosely at best, and a team quoting one number as the other will draw the wrong conclusions. Related but distinct again is change failure rate, which counts how often changes fail rather than how long recovery takes.

Why it matters

A red pipeline on a shared branch blocks everyone waiting on green: merges queue up, deploys stall, and the cost multiplies by team size — which is why the cost of failed builds is dominated by engineer wait time, not compute. A rising pipeline MTTR usually means failures are getting harder to diagnose, and the repositories dragging the mean upward are the place to look first.

TrimCI computes pipeline MTTR in its PDF reports from synced run history — mean, median and p95 across the window's red-to-green episodes, plus the longest red streaks and the pipelines still red when the window closed, each identified by repository, pipeline and branch — next to failure rates and error fingerprints, as part of CI pipeline failure analysis — so "how long do we stay broken" becomes a number instead of a feeling.