The Fight Against Flaky Tests — When Unreliable Tests Erode Trust

What “it fails sometimes” takes away

The previous article, Coverage and Mutation Testing, dealt with how to measure “test quality.” But no matter how high-quality the tests you write, if those tests fail capriciously, the suite loses its most important asset: trust.

A flaky test is one that passes or fails from run to run even though neither the code nor the test code changed at all. A failure from a real bug is reproducible; a flaky failure shows up as “ran it again and it passed.” The cause of the failure is neither a defect in the product code nor an error in the test logic — that is the heart of what makes flaky tests so insidious.

At a glance it looks like “a trivial problem where it just fails sometimes.” But its real harm quietly erodes the very foundation of the practice of testing.

Why it’s harmful

The harm of flaky tests hides in places that don’t show up in numbers.

• Erosion of trust: as the experience “even when it goes red, re-run it and it passes” accumulates, trust in the CI failure signal itself is lost
• The “boy who cried wolf” effect: even when a real regression happens, it’s dismissed as “probably just flaky,” and the bug leaks into production
• Slower development: time is drained triaging whether a failure is a real bug or flaky. The “normalized red” right before a release especially delays deploys
• Psychological cost: repeatedly seeing failures of unknown cause saps developers’ motivation to test

The value of tests is that they give “the confidence to change without fear.” Flaky tests erode that confidence from behind.

A taxonomy of causes

The causes of flakiness are diverse, but can be broadly organized as follows.

Category	Typical example
Non-determinism	Unseeded randomness, reliance on order-undefined collections like HashMap, floating-point rounding
Time / timeout dependence	Logic that depends on the current time, timeouts too short for CI load, tests crossing a date boundary (midnight, month-end)
Concurrency / race conditions	Asserting without waiting for async completion, depending on inter-thread execution order
Order dependence	One test implicitly creates the state another test assumes; fails when run order changes
Shared state between tests	Forgetting to clean up global variables, static fields, or shared DB records
External dependencies	Real network calls to external APIs, subject to the other side’s outages and latency
Resource exhaustion	CI memory/CPU shortage, port contention or disk shortage from parallel runs

These often appear not singly but in combination. A test with “insufficient waiting for async completion,” for example, fails only on days when the CI environment’s CPU load is high — “concurrency” and “resource exhaustion” intertwined.

Detection — rerun diff and flakiness rate

The most basic way to find flaky tests is to run the same test multiple times against the same commit and pick out the ones whose results waver.

• Rerun diff method: re-run a failing test a few to a few dozen times with no changes; if the result ever differs, mark it “flaky”
• Flakiness rate measurement: continuously record the ratio of “inconsistent results” over runs in a period, and auto-flag tests that exceed a threshold
• Statistical signal: mechanically extract from CI logs the event “both pass and fail observed for the same commit hash”

More and more CI systems (GitHub Actions, Jenkins, Buildkite) and test-reporting tools carry such rerun/flake-detection features, and automating detection itself is becoming established.

Google’s large-scale measurement

As a primary source showing that flaky tests are not “the carelessness of individual projects” but a challenge structurally attendant to large-scale development, the numbers Google published on its testing blog in 2016 are often cited.

• About 16% of all tests show flakiness at some level (more than one in seven tests engineers write can fail spuriously)
• Across all test runs, about 1.5% of results are judged “flaky”
• About 84% of pass-to-fail transitions are due to flaky tests, which greatly raises the cost of investigating “is this new failure real or not”

Google further notes that efforts to remove flaky tests and efforts that newly introduce them proceed at roughly the same pace — a structural state of “continuing to coexist with a certain flake rate.”

auto-wait — a structural countermeasure for E2E, the biggest breeding ground

As we saw in The Generations of E2E and Browser Testing, flakiness arising from timing — UI render completion, async communication, animation end — is especially common in the E2E layer. In the old generation (Selenium, etc.), fixed-time sleep was heavily used, and this itself was a breeding ground: “too short and it fails, too long and it’s slow.”

New-generation tools like Cypress and Playwright ship auto-wait as standard, automatically polling until an element is actually actionable (visible, animation finished, not disabled). This reduces reliance on fixed sleep and structurally mitigates one of the main sources of flakiness.

Quarantine and retry — the pros and cons of first aid

There are two realistic operations for a test found to be flaky. Both are “coexistence measures,” not “solutions.”

Quarantine temporarily removes a flaky test from CI’s pass/fail decision and keeps running it in a separate track. It has the merit of not blocking the mainline CI, preserving development speed — but what was supposed to be “temporary” easily gets neglected and made permanent. For the quarantined period, that test loses its power to detect a real regression. If the quarantine list grows without limit, the effective coverage of the whole suite dwindles.

Retry automatically re-runs a failing test a few times and treats it as passing if it succeeds even once. It is easy to implement and immediately effective — but it risks masking a real regression (a timing bug on the product side dismissed as “it passed on retry, so no problem”). Retry counts inflate and run time grows.

Neither works unless paired with ownership — who fixes it, by when — or it ends as merely institutionalized “looking the other way.”

Toward eradication — flakiness as a design problem

The essential remedy is to identify the cause along the taxonomy and remove it at the design level.

• Make it hermetic: have the test itself control all the state it needs (time, randomness, external dependencies) (see hermetic tests in Testing at Scale)
• Fix time and randomness: stop depending directly on the system clock; replace with injectable time sources and random seeds
• Test independence: have each test set up and tear down the data/state it needs self-containedly, severing implicit order dependence
• Explicit waiting for async completion: instead of sleep, poll for the target state change, or await a callback/Promise completion
• Mock external dependencies: replace real network calls with test doubles, decoupling from external services’ availability and latency

Much of flaky testing can be understood as design problems (implicit dependencies, missing boundaries) surfacing as symptoms.

The stage called CI/CD

Flakiness hurts most on the CI/CD pipeline, where tests keep running automatically. A modern pipeline runs tests in stages — unit → integration → E2E — placing the fast and cheap first and the slow and expensive last (fail-fast). As test counts grow, parallelization and sharding hold run time down, and Test Impact Analysis, which selects only the tests relevant to a change, cuts waste.

In this context, once “passing tests before merge” is a gate, flaky tests destabilize the gate itself. That is precisely why operations like flake detection, quarantine, and retry are built in as machinery to preserve CI/CD reliability. Test automation creates value only on the premise of test reliability.

Summary

Flaky tests are not “a trivial problem where they just fail sometimes” but a structural challenge that quietly erodes trust in the whole suite. The causes span non-determinism, time dependence, concurrency, shared state, and external dependencies, and as Google’s measurements show, they occur at a scale that can’t be ignored in large shops. Quarantine and retry are realistic operations that ease the symptoms, but the root solution requires securing independence and hermeticity at the design level.

Every test we’ve seen so far has been a practice of “run it and check.” But testing by execution has a fundamental limit — Dijkstra’s warning from the first article: “testing can show the presence of bugs, but not their absence.” In the next article we take up formal methods, which challenge this limit from another side.

References

• John Micco, “Flaky Tests at Google and How We Mitigate Them” (Google Testing Blog, 2016)
• Google Testing Blog, “Where do our flaky tests come from?” (2017)
• Official documentation of E2E tools (Cypress / Playwright) on auto-wait / retry