Why is my CI pipeline slow?

A slow CI pipeline is slow for one of two reasons: jobs wait too long for a runner to pick them up (queue time), or the work takes too long once it starts (execution time). Everything else — missing caches, serial job graphs, undersized runners, bloated images, retry habits — is a sub-cause of one of those two. Diagnosing which half dominates, and for which jobs, is the entire first step. A fix aimed at the wrong half changes nothing.

That corollary is worth sitting with, because most "speed up your CI" advice skips diagnosis entirely. Adding a dependency cache to a pipeline that spends four minutes queued and ninety seconds executing will not move the number your developers feel. This guide is a diagnosis path: split wall-clock time into queue and execution, read percentiles instead of averages, then match the dominant symptom to a ranked list of causes — in that order. Only then spend an afternoon optimizing.

Queue time vs. execution time

The duration a developer experiences is not "how long the jobs ran." It is:

time to feedback = queue time + execution time + the gaps between dependent jobs

Queue time is the span between a job becoming eligible to run and a runner actually starting it. Execution time is the span from start to finish. Both GitHub Actions and GitLab CI expose the timestamps needed to separate them — job creation, start, and completion times (as of 2026-07-28: GitHub workflow jobs, GitLab jobs API, which also ships a ready-made queued_duration). One caveat: a GitHub workflow run carries no completion timestamp — only created_at, run_started_at and updated_at (workflow runs) — so run-level durations there are an upper bound that still includes the queue wait, while GitLab's pipeline duration excludes it. The split needs no instrumentation, just careful arithmetic on data you already have.

The reason to start here is that the two halves have disjoint fixes:

• Queue-dominated pipelines have a capacity problem. Concurrency limits, a fixed pool of busy self-hosted runners, peak-hour contention, or scarce specialty runners (large machines, GPU, macOS) are the usual culprits. No amount of caching, test sharding, or image slimming will help, because the work is not the bottleneck — the wait for a machine is.
• Execution-dominated pipelines have a work problem. The runner starts promptly and then grinds through uncached installs, serial jobs, heavyweight images, or a test suite that has quietly doubled. Buying more runners here mostly buys you more idle runners.

One subtlety that hides real waiting: queue time is paid per job, not per pipeline. A pipeline with six sequential stages can pay the queue toll six times, and most provider UIs surface only the headline duration. If your jobs individually look fast but the pipeline feels slow, sum the inter-job gaps before blaming the jobs.

Why averages lie

The second diagnostic habit is refusing to reason from the mean. An "average pipeline duration" blends runs that have nothing in common: a docs-only change that skipped the test job, a cache-hit run, a cold-cache run after a lockfile bump, a run that sat in queue behind a nightly batch. The blended number describes none of them.

Duration percentiles fix this. P50 — the median — is what a typical run feels like. P95 is the run that ruins an afternoon, and it is the number developer perception actually tracks, because people remember waiting 25 minutes far more vividly than the eight-minute runs in between. A pipeline can average nine minutes while its P95 sits above twenty; the team calling that pipeline "slow" is right, and the average is wrong.

The gap between P50 and P95 is itself a diagnosis. A narrow gap means the pipeline is uniformly slow — look at the work itself. A wide gap means the pipeline is sometimes slow, which points at variance sources: cache misses, runner contention at busy hours, retried jobs, or a test suite whose duration depends on which shard a flaky test lands in.

Two hygiene rules before you compute anything. First, measure per job as well as per pipeline — a healthy P95 at the pipeline level can hide one job whose P95 has doubled, masked by parallel siblings. Second, exclude noise runs: a wave of superseded runs auto-cancelled by newer pushes drags averages down and tells you nothing about the experience of a developer who waited for a real answer. (The distinction matters enough that we keep a separate note on cancelled vs. failed runs.)

The usual suspects

With queue and execution separated and percentiles in hand, the cause usually identifies itself. These five account for most slow pipelines, ranked roughly by how often they dominate — each with the signature it leaves in the data, which is the point of diagnosing first.

No dependency caching

The most common execution-time tax is re-downloading and re-installing the same dependencies on every run. The signature is a setup phase that costs the same fixed minutes whether the commit changed one line or one hundred, visible as job durations that are high but suspiciously stable. The fix is standard — cache the package manager's store keyed on the lockfile, cache container build layers — but measure after enabling it: a cache that saves forty seconds of installs while spending a minute restoring a multi-gigabyte archive is a net loss. Cache hit rate and restore time are part of the diagnosis, not an afterthought.

Serial jobs that could run in parallel

If pipeline wall-clock time roughly equals the sum of its job durations, nothing meaningful runs in parallel. Lint, type checks, unit tests, and builds are usually independent; a test suite can often shard across runners. The honest caveat: parallelism trades money and queue exposure for speed. Fanning one job out into eight means paying eight queue tolls and eight runners' minutes, which is a fine trade when it converts a thirty-minute critical path into eight — and a poor one when queue time is already your bottleneck. This is why the queue/execution split comes first.

Runner under-provisioning

The queue-dominated pattern: execution times are steady, but queue P95 balloons — typically at the team's busiest hours, while the same pipeline starts instantly at 7 a.m. On self-hosted fleets the cause is a fixed pool sized for average load rather than peak; on hosted runners it is usually a concurrency ceiling or contention for scarce runner classes. Compare queue percentiles by hour of day: a flat profile suggests a configuration limit, a peaked one suggests capacity.

If the diagnosis genuinely lands on hardware — CPU-bound builds on small machines — faster-runner vendors exist to sell you exactly that, and switching runners is sometimes the correct fix. TrimCI is a measurement layer, not a runner vendor, so the two are complementary rather than competing; the TrimCI vs. Blacksmith comparison walks through where each fits.

Bloated images and slow checkout

A relative of the caching problem: the job spends its first minutes pulling a multi-gigabyte container image, cloning full repository history, or installing a toolchain that could have been baked into the image. The signature is a long gap between job start and the first command that matters to the build. Slimmer base images, prebuilt images carrying the toolchain, and shallow clones each shave a fixed cost off every single run — small per run, large multiplied by thousands of runs a month.

Retries hiding flakiness

The sneakiest cause, because it makes pipelines slow while keeping them green. When flaky tests are managed with auto-retry or a re-run habit, every affected pipeline silently runs some of its work twice — and the developer's time to feedback includes both attempts plus the human delay of noticing the failure and clicking re-run. A pipeline that "takes 12 minutes" but fails spuriously one run in five has a real-world feedback time far above 12 minutes, and none of it shows up in the duration column. The compute half of this cost is the retry tax; on GitHub Actions it is directly measurable from attempt counts in run metadata, while GitLab CI has no run-level attempt counter — both the built-in retry: keyword and a manual retry create new job instances inside the same pipeline, which keeps its ID (as of 2026-07-28, GitLab pipelines API, GitLab jobs docs), so extra attempts never surface as extra pipelines. The pattern shows up there as retried jobs within a pipeline, or as fail-then-pass alternation on the same branch when someone re-triggers the pipeline from scratch.

Failures deserve their own mention here: the slowest pipeline is the one you had to re-run after reading the logs. If a meaningful share of your runs fail and get retried, duration tuning is the wrong starting point — failure analysis is.

How long should a pipeline take

Honestly: it depends, and any page giving you one universal number is guessing. Pipeline duration budgets follow from what the pipeline blocks. A pull-request check blocks a human who is deciding whether to context-switch away; a merge pipeline blocks a deploy train; a nightly regression suite blocks nobody in real time. The same twenty-five minutes is unremarkable for the third and corrosive for the first.

The old continuous-integration rule of thumb — keep the build under ten minutes — has survived decades of tooling changes because it is really a claim about human attention: under roughly ten minutes, a developer plausibly waits; beyond it, they context-switch, and the effective cost of the pipeline becomes the interruption, not the minutes. That reasoning, not any benchmark, is what the reference points below encode. Treat them as working targets to argue with, not industry standards, and judge them on P95 over real runs — not the average, and not a lucky Tuesday sample.

The queue-time row deserves emphasis because it is the one teams skip. Queue time buys you nothing — it is pure waiting — so its budget is relative, not absolute: when the P95 queue wait is a meaningful fraction of execution time, you have a capacity problem regardless of how optimized the jobs are.

The measure-then-fix loop

Generic optimization advice is interchangeable precisely because it omits the loop that makes any of it work. The loop is short:

1. Baseline over a real window. Two to four weeks of runs, enough for a stable P95. A single day's sample reflects that day's luck — one dependency bump or one busy afternoon distorts everything.
2. Split queue from execution. This tells you whether you are buying capacity or removing work. At pipeline level it is a subtraction on run timestamps; per job it needs your provider's job-level creation timestamps, which not every tool stores.
3. Find the critical path. The pipeline is as slow as its longest dependent chain, not its longest job list. Optimizing a job that runs in parallel with a slower one changes nothing a human can feel.
4. Change one thing. A cache, a shard, a runner size — one variable, or you will not know what worked.
5. Re-measure the same percentile over a comparable window. P95 against P95, busy weeks against busy weeks. Declaring victory off the average after a quiet week is how "we sped up CI" becomes folklore instead of fact.
6. Keep watching. Duration regressions arrive in five-second increments — a new dependency here, three added tests there — and no single commit ever looks guilty. A monthly glance at P95 trend and retried-run share catches the creep while it is still cheap to reverse.

It is worth being explicit about why the loop pays for itself: a slow pipeline bills you twice. Once in compute minutes — a cost you can bound with the numbers on your provider invoice — and once in engineer waiting and context-switching, which is almost always the larger figure. The cost of failed builds breaks down that second bill, the calculator on our landing page puts rough numbers on it for your team size, and the same run history that diagnoses slowness also drives spend, which is the subject of reducing GitHub Actions costs.

Start from data you already have

Everything this guide asks for — queue timestamps, durations, attempt counts, outcomes — lives in your CI provider's ordinary run and job metadata. You do not need to instrument builds, add tracing, or grant anything access to your source code to answer "why is my pipeline slow"; you need the arithmetic done consistently over enough history to trust the percentiles.

That is the part TrimCI automates. It syncs run and job metadata from GitHub Actions and GitLab CI (gitlab.com and self-hosted CE/EE 14+ alike, when the instance is reachable over HTTPS), then computes queue time, P50/P95 duration percentiles, retry tax, and failure costs across the repositories you select (percentiles are broken out per provider, since durations are not comparable across GitHub and GitLab) — from pipeline metadata alone, with zero access to your code. The honest boundaries: durations and queue time are measured at pipeline-run granularity — job-level rows are collected for failed runs, not for the green-but-slow ones — so it will point at the pipeline that blew the budget rather than the job or the individual test inside it; data arrives by API polling (hourly on paid plans, daily on the free one) rather than real-time webhooks; and analysis windows extend to 60 days depending on plan — the per-plan sync cadences and history windows are listed on the pricing page. For diagnosing why a pipeline is slow — and proving whether last month's fix actually worked — that is exactly the data the loop above needs.