Testing at Scale — Millions of Tests, and the Design That Sustains Them

Running a few hundred tests in a team of a few dozen people is one thing. Running millions of tests against a single, enormous codebase (a monorepo) that tens of thousands of engineers commit to tens of thousands of times a day is another — the problem changes in kind, not just in size. Organizations like Google and Meta have kept testing at this scale working by specializing their build systems, execution infrastructure, and flaky-test management. But execution infrastructure isn’t the whole story: testable design, a seemingly unrelated topic, answers the same underlying question.

The Monorepo Premise — Google’s and Meta’s Giant Test-Suite Culture

Google and Meta are both known for running nearly their entire business out of a single monorepo. Google’s internal build system Blaze (used internally since around 2004, open-sourced as Bazel in 2015) and Meta’s Buck (open-sourced in 2013, superseded by the Rust-based Buck2 in 2023) share the same philosophy: treat the entire monorepo as a single build graph, with test targets embedded in it.

At this scale, the idea of “rerun every test on every change” simply becomes unworkable. Re-running millions of tests on every commit would clog the CI queue and delay feedback by hours. The core of monorepo operations is building the “run only the tests relevant to the change, reliably leveraging the cache” capability directly into the build system.

Hermetic Tests — The Foundation Everything Else Rests On

The central concept underpinning the reliability of large-scale test suites is the hermetic test. This is a principle central to the book Software Engineering at Google, defined as follows:

A test should contain within itself all the information needed to set itself up, run, and tear itself down.

A hermetic test doesn’t depend on an external shared database, a real network, or state specific to the execution environment. If disk access is required, it’s isolated from the environment — for instance, by using an in-memory filesystem implementation instead of the real disk. Hermeticity structurally eliminates the major causes of flakiness (external dependencies, networks, shared state) covered in the earlier article The Fight Against Flaky Tests, and it’s also the precondition that lets caching (discussed below) work safely.

Test Size Classification — Small / Medium / Large

Google classifies tests not by “what the code does” but by “how it’s allowed to run and what it’s permitted to touch.”

Class	Execution constraints	Typical runtime
small	single process, single thread; no network or disk I/O	tens of milliseconds+
medium	confined to a single machine; no network calls beyond localhost	a few seconds
large	may span multiple machines	tens of seconds to minutes

Size is independent of the “unit test vs. integration test” axis — it’s determined purely by the environmental constraints imposed at runtime. Google’s commonly cited target distribution is roughly 80% small, 15% medium, 5% large, which can be understood as re-deriving the spirit of the test pyramid (covered in The Shapes of Tests) along the axis of monorepo execution constraints.

Remote Caching and Test Impact Analysis

A core capability shared by Bazel and Buck2 is remote caching. Every input to a build or test (source code, dependency versions, compiler flags) is hashed, and if a result already exists in the cache for that same input hash, it’s reused rather than recomputed. Hermeticity, from the previous section, is exactly what makes this cache safe to trust. Buck2 is the Rust-based successor to Buck (originally written in Java) and, like Bazel, adopts Starlark (a Python-like configuration language) — Blaze, Bazel, and Buck2 are commonly described as tools from the same lineage.

Alongside caching, the other core technique is test impact analysis. The static approach walks the build graph to mechanically identify which test targets could be affected by a changed file (the query capability in Bazel and Buck2). The dynamic, statistical approach trains a machine-learning model on historical execution data to learn the correlation between “patterns of code changes” and “tests that actually failed.” Meta’s Predictive Test Selection, published in 2018, is a well-known example: by narrowing down which tests to run, it reportedly halved the infrastructure cost of test execution while continuing to catch over 95% of individual test failures and over 99.9% of defect-introducing code changes. The same research is also cited for modeling test flakiness into its predictions.

Distributed Execution and the Build Graph

Millions of tests are simply beyond what sequential execution on a single machine can handle in reasonable time. Bazel and Buck2 mechanically guarantee, from the build graph, that test targets with no dependency relationship can run in parallel, and distribute them across many worker machines via remote execution. The practice of “running only the tests affected by a change, distributed and parallelized across many machines” works precisely because caching, test impact analysis, and distributed execution all share the same underlying abstraction — the build graph. This parallelization, too, presupposes hermeticity (that tests share no state with one another).

Scale Increases the Absolute Count of Flaky Tests

The larger the scale, the larger the absolute number of flaky tests discussed in the earlier article. According to a 2016 report published by Google (John Micco and colleagues), about 16% of internal tests exhibited flaky behavior at some level, while tests that were flaky continuously (failing inconsistently on nearly every run) were around 1.5%. This isn’t a sign of individual teams’ negligence but a structural challenge that comes with scale, and large organizations maintain dedicated infrastructure for it: continuous re-execution to detect pass/fail variance, automatic flagging of tests whose flake rate crosses a threshold, and logic that excludes known flaky failures from CI pass/fail decisions.

Other challenges surface more as scale grows — cache consistency collapses if a non-hermetic test slips in, dependency graph bloat as the monorepo grows the build graph itself, diluted ownership as it becomes unclear who’s responsible for a given test, and a divergence between environments optimized for remote caching and remote execution versus the local development experience of individual engineers.

Execution Infrastructure Isn’t Enough — the Idea of a Seam

Everything above rests on one foundation: that individual tests are hermetic — that they don’t depend on a real database or network, and that their dependencies can be substituted. But if the code isn’t written to be substitutable in the first place, hermeticity is a pipe dream. This is where the execution-infrastructure story connects to the design story.

In Working Effectively with Legacy Code (2004), Michael Feathers systematized techniques for adding tests to legacy code that has none, placing the concept of a seam at its core. A seam is “a place where you can alter behavior without editing the code in that place” — a constructor argument that lets you substitute a dependent object (an object seam), a dependency that can be swapped at link time (a link seam), or substitution via macros (a preprocessor seam). Code without seams — everything hard-wired with new, directly dependent on static methods or global state — has no entry point for tests, and no entry point for hermeticity either.

Dependency Injection and Hexagonal Architecture

The most common way to create a seam is dependency injection (DI).

// Without DI: dependency created internally → tests need a real SMTP server
class OrderService {
    private EmailSender sender = new SmtpEmailSender();
}
// With DI: dependency injected externally → tests can substitute a Fake
class OrderService {
    OrderService(EmailSender sender) { this.sender = sender; }
}

Hexagonal architecture (ports and adapters) was proposed by Alistair Cockburn in 2005. Core logic sits at the center; interaction with the outside world happens through “ports” (interfaces), and connections to concrete technology are handled by “adapters.” Cockburn’s motivation was to let an application “be driven equally by a user, a program, automated tests, or a batch script, and be developed and tested apart from the eventual devices and databases.” This architecture was later reworked into Clean Architecture (Robert C. Martin) and Onion Architecture (Jeffrey Palermo), all sharing the idea of “pointing dependencies inward, toward the core.”

Humble Object, and Functional Core / Imperative Shell

The Humble Object pattern is the idea of keeping the hard-to-test part of a system as empty as possible. Its origin traces back to Feathers’s 2002 paper “The Humble Dialog Box,” which proposed moving GUI logic into a separate class, leaving the dialog itself as a “humble class that only displays.” The idea was generalized in Steve Freeman and Nat Pryce’s Growing Object-Oriented Software, Guided by Tests (2009, commonly called the GOOS book) into a general pattern applicable to any “hard-to-test boundary” — database access layers, external API calls, and beyond. The “keep the View thin” design in MVP and MVVM is one instantiation of it.

Functional core, imperative shell is a design style popularized by Gary Bernhardt in his 2012 conference talk “Boundaries” (SCNA 2012), separating a “core” made entirely of pure functions from a thin “shell” that handles I/O and side effects. The core needs no mocks or stubs and can be tested fast and deterministically. Bernhardt has acknowledged that this style drew inspiration from Cockburn’s ports and adapters — DI, hexagonal architecture, Humble Object, and functional core are all different expressions of the same underlying principle: push side effects to the boundary, and keep logic pure.

Why Global State Breaks Tests, and Friction as a Warning Sign

Singletons and global variables make testing hard for four reasons: order dependence from shared state between tests, the impossibility of substitution, conflicts under parallel execution, and implicit dependencies that don’t appear in a constructor. DI and hexagonal architecture can be described as ways of converting this “implicit dependency” into an “explicit dependency.”

The friction you feel while writing a test can be read as a design warning.

Friction	What it suggests
Setup is absurdly long	a class with too many responsibilities
The test is drowning in mocks	a mismatch in abstraction level
Tests only pass in a specific order	a hidden dependency on global state
Results change due to time or randomness	side effects not pushed to a boundary

Maintainability, the Other Battleground — Test Smells and DAMP

Tests at the scale of millions become a high-maintenance liability if written poorly. In his 2007 book xUnit Test Patterns: Refactoring Test Code, Gerard Meszaros catalogued 68 patterns along with a taxonomy of test smells.

Test smell	Description
Fragile Test	breaks from trivial changes (refactorings that don’t affect behavior, etc.)
Mystery Guest	data that determines the test’s outcome is hidden outside the test (shared fixtures, existing DB records)
Assertion Roulette	many assertions crammed into one test, so you can’t tell which one failed
General Fixture	a shared fixture reused across tests, carrying extra data none of them individually need
Test Run War	multiple tests (or processes running in parallel) fight over the same shared resource (a DB, a file, a port)

Against DRY (Don’t Repeat Yourself), widely shared in production code, the testing world has settled on the counter-slogan DAMP (Descriptive And Meaningful Phrases). Pushing shared setup too far means that understanding one test requires tracing back through multiple helper methods — which is exactly what produces a Mystery Guest. In practice, the common understanding is that the two coexist in most situations: eliminate meaningless duplication, but don’t over-abstract away the structure that communicates a test’s intent.

Overusing mocks also damages maintainability. Over-specification — a test that verifies call order or call count — breaks under harmless refactorings of internal implementation. The widely recommended discipline is to “mock only the direct collaborators of what you’re testing, and use the real thing for value objects.” For generating test data, there are two well-known patterns: Object Mother, which prepares canonical examples, and Test Data Builder, which carries defaults while letting you override only the attributes that matter for a given test. The latter has become more widely adopted in recent years because it more easily avoids General Fixture.

Scale and Design Are, in Fact, Answering the Same Question

At first glance, monorepo execution infrastructure (hermeticity, caching, test impact analysis) and individual class design (DI, hexagonal architecture, Humble Object) look like separate topics. But hermeticity is only achievable once you have a design where dependencies can be substituted; small tests can make up 80% of the suite only because side effects have been pushed to a boundary; and millions of tests can remain reviewable and ownable by humans only because they’re readable — written with DAMP in mind. What testing at scale teaches us is a single point: the infrastructure question of “how do we run it” and the design question of “how do we write it” become inseparable the larger the scale grows.

From This Article to the Next

The execution infrastructure and design that sustain millions of tests were both engineering answers humans built up over time. But in the 2020s, AI has begun to take part in generating, repairing, and evaluating tests too. In the final article, we look at how generative AI is changing testing — and what it isn’t changing.