Defining a Specification as a Test Suite — "The Spec Is: It Passes roast"

Last time I ended on my own language:

If the oracle itself is wrong, all five are wrong together.

This instalment is about how Perl 6 handled that question. On what basis do we say something is correct as Perl 6?

Three Layers of Design Document

Part 3 got as far as Larry Wall digesting the 361 RFCs. The output was the Apocalypses. In practice, there were three kinds of document.

Document	Author	Role	Metaphor
Apocalypse	Larry Wall	Design decisions and their reasons: why	Revelation
Exegesis	Damian Conway	Explains an Apocalypse with examples	Commentary
Synopsis	Various	A summary usable as a spec; what implementers consult	Summary

The names are all biblical, which is very much Perl culture.

Apocalypse numbers match the chapters of Programming Perl, the Camel Book: each answers “what happens to chapter N in Perl 6.”

Number	Subject
A1	The Ugly, the Bad, and the Good — overall direction
A2	Basic data types, sigils, literals
A3	Operators
A4	Control structures
A5	Pattern matching — the regex redesign. The single most consequential one
A6	Subroutines — signatures, multiple dispatch
A12	Objects

Not all of them were written. That is why the numbers skip.

The Synopses Became the Working Spec

What implementers actually consulted day to day was not the Apocalypses but the Synopses (S01–S32).

The reason is plain. An Apocalypse is long prose explaining why. It is not suited to being looked things up in. The Synopses were organised to be consulted, and they kept being updated.

Number	Subject
S02	Lexical structure, data types, sigils
S03	Operators (including meta-operators)
S05	Regexes and grammars
S06	Subroutines and multiple dispatch
S12	Objects
S14	Roles and types
S17	Concurrency
S32	Standard library

That is the era of the prose specification.

Three Problems With a Prose Specification

And that form has problems of its own.

1. Interpretations Diverge

When implementers read differently, implementations disagree. As part 5 covered, Perl 6 had several implementations in this period: Pugs, the Parrot-based one, and later Niecza on .NET.

Reading the same Synopsis and implementing different behaviour happens. Prose contains no procedure for deciding which is right.

2. It Drifts From the Implementation

If the document stops being updated, nobody notices.

This is the general problem with design documents. Documents do not execute. So when they are wrong, or stale, nothing turns red. It is the same structure as “unverified decisions accumulate, unverified” from part 5.

3. Conformance Cannot Be Checked

This is the most practical problem.

When an implementation claims to “conform to Perl 6,” there is no procedure for verifying the claim. Someone has to read the prose and decide.

roast — Moving the Spec Into Tests

So Perl 6 moved authority to a test suite called roast: the Repository Of All Spec Tests.

Its origin, as part 5 described, is the practice Audrey Tang started with Pugs — writing a test into the official suite for every feature implemented.

And eventually this was declared:

The specification is: it passes roast.

What “Being 6.c” Means

The 6.c release of 2015 — part 8’s subject — was defined by freezing a particular commit of roast.

So “conforms to 6.c” means “passes roast as of 6.c.”

That is startlingly concrete. The problem of interpreting prose disappears.

• When an implementation claims 6.c conformance, that claim is mechanically checkable
• A new version is defined as a new frozen point of roast
• Old behaviour is reproducible, as long as the old roast is preserved

And the state described in part 12 — 6.d as default, 6.e in progress — simply means there are two frozen points and a third is being built.

What Was Gained, and What Was Given Up

Gained	Given up
Conformance is mechanically checkable	Behaviour not written into a test stays unspecified
Spec and implementation cannot drift apart	Tests do not tell you why
Versions can be frozen (6.c / 6.d)	Tests can quietly pin an implementation’s convenience

The first row on the right matters most. Here is this series’ eighth pattern.

Prose speaks vaguely about everything; tests speak precisely about some things.

A prose specification says something about the whole language — vaguely. A test specification decides exactly, where it is written. And says nothing at all where it is not.

This is not a ranking. It is that the shape of the silence differs.

• Prose falls silent in a way you can see: “this part is vague”
• Tests fall silent in a way nobody notices: “there is no test here”

Which is why, even after moving to roast, Raku did not abandon prose. The Synopses remain as historical documents, and the language documentation (docs.raku.org) carries the practical explanation. What moved to roast is authority.

How Other Languages Do It

This decision is not unique to Perl 6.

Language	Authoritative spec
Raku	roast (a test suite)
ECMAScript	ISO/ECMA prose specification + test262 (shared tests)
Ruby	The CRuby implementation + ruby/spec
C / C++	ISO standards (prose)
Go	A language specification (prose) + a single reference implementation

ECMAScript is the closest. It has both prose and test262, and “conformance” is measured by test262.

What makes Raku unusual is that it kept the spec separate from the implementation through a long stretch of having only one. As part 6 covered, from the moment Niecza stopped, Rakudo was the only usable Raku.

Normally, with one implementation, “the implementation is the spec” suffices. That is Ruby’s position (CRuby is the de facto spec). Raku maintained roast anyway — partly as an inheritance from the era of several implementations, and partly because it is a breakwater that matters precisely once you are down to one.

And as the end of this section describes, in 2026 that inheritance paid off.

With a single implementation, an implementation bug can settle in as “the language’s specification.” With roast, you can ask: is that written in a test?

Back to Last Time’s Question

Part 6 ended here:

If the oracle itself is wrong, all five are wrong together.

roast solves that halfway.

The part it solves: the specification lives somewhere independent of the implementation. How Rakudo behaves and how Raku ought to behave are in different repositories. An implementation bug does not automatically become the spec.

The part it does not: it can say nothing about what is not written in roast. In terms of my own parity setup: the region where the interpreter is wrong and nobody has written a test for that behaviour is still undefended.

Perl 6 itself plugged that hole another way. It had several implementations. When Pugs, the Parrot-based one, and Niecza all run the same roast, behaviour present in only one of them is either a hole in the spec or a bug in the implementation — and you can tell.

A second implementation produces a kind of bug the first one cannot notice.

Part 6 said the second user never came. From the standpoint of verifying a specification, the departure of the second implementation is the same kind of loss.

From the moment Niecza stopped, Raku maintained roast with a single implementation for over a decade. Reasonable — but one means of verifying how much of the specification roast actually captures had been removed.

And in 2026, a second one came back

As this is being written, in 2026, two implementations other than Rakudo are running.

Implementation	Language	Approach	Roast (files fully passing, of 1,464)
mutsu	Rust	Bytecode VM	1,433 (97.9%)
Raku++ (rakupp)	C++17, no third-party deps	Hand-written lexer and parser, tree-walking evaluator; also compiles to a native binary	676 (46%)

Both measure themselves with roast as the denominator. “Conforms to Raku” being a number rather than a claim is possible because the specification was moved into a test suite. This instalment’s subject is doing its work right here.

⚠️ Figures checked on 18 September 2026. Both projects move constantly; mutsu’s README states outright that compatibility improves daily. Re-measure before quoting.

So the worry in this section is partly resolved. The means of re-asking how much of the specification roast captures has returned after a decade.

A second implementation produces a kind of bug the first one cannot notice.

That pattern is usable again.

---

Next (part 8): Christmas. On 25 December 2015, fifteen years after the announcement, the first stable release arrives. What were those fifteen years? Time to add it up.