Promoting Regular Expressions to a Language Feature — Grammars, the Thing Only Raku Has

From here, Raku as a language.

Part 4 laid out nine things. If you pick only one, pick this. The thing Raku has that other languages don’t — grammars.

The Starting Point — the Notation Had Saturated

Perl 5’s regular expressions became the de facto industry standard. Exported as PCRE, they are still the basis of most languages’ regex syntax.

And the notation had saturated.

There were no symbols left to add, so every new feature had to invent something beginning with (?.

(?:...)     # non-capturing group
(?=...)     # lookahead
(?!...)     # negative lookahead
(?<=...)    # lookbehind
(?<name>…)  # named capture
(?#...)     # comment
(?{...})    # code execution

All of them start with (?. Adding anything new means hunting for an unused symbol to put after (?.

As notation design, that is a dead end. Apocalypse 5 — the most consequential of the design documents mentioned in part 7 — went after exactly this.

Reallocating Notation by Frequency

Raku broke compatibility and reorganised. The main changes:

Change	Perl 5	Raku
Whitespace	Significant	Ignored by default (/x behaviour is the default)
Literal strings	Bare	Quoted with '...'
Grouping (non-capturing)	(?:...)	[...]
Character class	[abc]	<[abc]>
Lookahead	(?=...)	<?before ...>
Calling a named rule	(?&name)	<name>

Swapping the roles of [ ] and < > is the largest incompatibility.

Why? Frequency.

Non-capturing groups get written more often than character classes. Yet in Perl 5 the more frequent one was longer ((?:...)) and the less frequent one was shorter ([...]).

Raku inverted it. It re-matched the length of the notation to how often it is used.

Here is this series’ twelfth pattern.

The length of a notation is an allocation against frequency. Keep adding features and that allocation will inevitably go wrong.

And fixing the allocation requires breaking compatibility. It is the notation-level version of part 2’s “a successful language gets fixed in the shape of its success.”

if "2026-09-18" ~~ / ^ (\d ** 4) '-' (\d ** 2) '-' (\d ** 2) $ / {
    say "year=$0 month=$1 day=$2";
}

Whitespace is free, so it reads well without an /x modifier. '-' is quoted, so you can see at a glance that it is a literal.

The Key to Promotion — regex / token / rule

Now the substance.

Raku lets you name a pattern and reuse it. And there are three declarators.

Declarator	Backtracks	Handles whitespace
regex	Yes	No
token	No (ratchet)	No
rule	No	Yes (inserts <.ws>)

This three-way distinction is the key that promoted regexes into a parser.

Why? Writing a parser requires two decisions.

1. Do we backtrack? Lexing normally does not — a token, once decided, is decided
2. Do we skip whitespace? Not at the lexical level; yes at the syntactic level

Perl 5’s regexes had nowhere to declare either. They always backtrack, and whitespace is always significant. So writing a parser meant hand-coding both, every time.

Raku made you choose with a declarator.

• token is the default choice: no backtracking, so it is fast and its behaviour is predictable
• rule is for grammars where whitespace may appear between tokens — i.e. ordinary programming languages

Lexing and parsing became writable in the same notation.

Grammars — a Parser as a Class

Named rules bundled together form a grammar.

grammar Calc {
    rule  TOP    { <expr> }
    rule  expr   { <term> +% <addop> }
    rule  term   { <factor> +% <mulop> }
    rule  factor { <number> | '(' <expr> ')' }
    token number { \d+ }
    token addop  { '+' | '-' }
    token mulop  { '*' | '/' }
}

say Calc.parse('1 + 2 * 3');

Things to notice:

• TOP is the start rule. Run it with .parse / .parsefile
• The result is a tree of Match objects
• +% means “one or more, separated by.” <term> +% <addop> is “terms separated by addops” — a run of binary operators without writing left recursion
• Leaves are token, structure is rule. The whitespace decision is visible in the declaration

Seven lines for a calculator grammar. And it reads. The structure is more directly visible than in a yacc grammar or a nest of parser combinators.

A Grammar Is a Class — So It Inherits

grammar is a kind of class, so it can be inherited.

grammar CalcWithPower is Calc {
    rule factor { <base> ['**' <exp>]? }
    ...
}

Inherit an existing grammar and override only some rules.

Most parser generators do not have this. In yacc or ANTLR, “base it on this grammar but change this one rule” is not something you write directly; you copy the grammar file and edit it.

Where this earns its keep in practice is dialects: SQL dialects, config-file extensions, custom markup notations. Write the base grammar once, inherit the delta per dialect.

Actions — Separating Structure From Meaning

A grammar only recognises structure. Meaning comes from an Actions class.

class CalcActions {
    method TOP($/)    { make $<expr>.made }
    method expr($/)   { make [+] $<term>.map(*.made) }
    method term($/)   { make [*] $<factor>.map(*.made) }
    method factor($/) { make $<number> ?? $<number>.made !! $<expr>.made }
    method number($/) { make +$/ }
}

say Calc.parse('1 + 2 * 3', actions => CalcActions.new).made;   # 7

• $/ is the current Match
• make sets “the result of this rule”
• .made retrieves a child’s result
• [+] and [*] are the reduce meta-operators from part 4, folding the list of <term> results (this Actions class is a simplified one handling only + and *; - and / are omitted)

Grammar and interpretation are separated.

So giving the same grammar different Actions builds different things: an evaluator, a formatter, a type checker, a syntax highlighter — one grammar suffices.

This matters as implementation. When a grammar definition is copied into several places, it will drift. If several processors can be built from one grammar, there is nothing to drift.

Raku Itself Is Written as a Grammar

Decisively: Raku’s own grammar is written as a Raku grammar.

Historically that was STD.pm6 — a document maintained by Larry Wall that described Perl 6’s grammar in Perl 6’s own grammar notation. It sat close to being an executable specification, and dedicated tooling could actually run it.

The consequence is that users can extend the grammar at compile time.

sub infix:<∈>($x, @set) { $x (elem) @set }
say 2 ∈ (1, 2, 3);     # True

Operators are defined under names like infix:<...> / prefix:<...> / postfix:<...> / circumfix:<...>, with associativity and precedence. The moment you define one, it is really syntax.

Part 4 said they “raised what gets added from features to rules.” Grammars are the deepest instance of that. The language provides the means of extending the language.

⚠️ But this property has a price. In RakuAST — the compiler rebuild covered in part 12 — the grammar’s structure changes, so extensions that relied on the old grammar need updating. A language that decides to open its grammar pays twice when it changes that grammar.

Compared With Other Parsing Tools

Tool	Position
Raku grammar	A language feature. Inheritable, meaning separated via Actions, usable at runtime
ANTLR / yacc / bison	External tools. A code-generation step is required
Parsec (Haskell) / nom (Rust)	Libraries (parser combinators). Written as host-language functions
Python’s re / Ruby’s Regexp	Regexes only. You cannot write a grammar

Raku’s position is closest to “parser combinators made into language syntax.”

What differs from Parsec is that it has dedicated notation, and that notation is a continuation of regular expressions. Someone who knows regexes can start writing a grammar immediately.

Part 5 listed “parser combinators are available” as one of the reasons Audrey Tang wrote Pugs in Haskell. Raku has that as a language feature rather than a library.

Other Languages Can Extend Their Syntax Too

Let me draw the line honestly here. Raku is not the only language that can extend its syntax. But the capability splits into three tiers, and different languages sit in different tiers.

Tier 1 — Defining operators (with precedence and associativity)

Not rare. One example predates Raku by thirty years.

Language	How
Prolog	op/3 (priority 1–1200, xfx/xfy/yfx…). Since the 1970s
Agda	Mixfix: if_then_else_. Arbitrary word order via _ positions
Swift	infix operator + precedencegroup. You can create new precedence groups
Haskell	infixl 6 <+>
Raku	infix:<∈> + is tighter / is looser / is equiv
Scala / OCaml / F#	Infix operators exist, but precedence is fixed by the first character

Tier 2 — Adding new syntactic forms (macros)

Also wide.

The Lisp family (Common Lisp / Scheme / Racket / Clojure), Rocq’s (formerly Coq) Notation, Rust’s macro_rules! and procedural macros, AST macros in Elixir / Julia / Nim, and Haskell’s Template Haskell with quasiquotation.

The shape of the constraint differs, though. Rust’s macros work on token trees and require balanced delimiters. You can add new syntactic forms, but you cannot change the tokeniser.

Tier 3 — Replacing the parser itself

This tier is narrow. Raku’s slangs live here.

Language	What it can do
Racket	#lang replaces the reader entirely. You can define a language whose surface syntax has nothing to do with S-expressions
Raku	Slangs. The grammar can be switched mid-parse (lexically scoped)
Seed7	A language designed around extending syntax and semantics
Forth	Immediate words. There is essentially no fixed syntax to begin with
OCaml (Camlp4/5)	A syntax-extension preprocessor. Legacy today
Perl 5	Source filters (textual substitution). Brute force, in practice

In this tier, Racket is arguably stronger than Raku. Because #lang replaces the reader wholesale, you can put a language on top of Racket that does not look like Racket at all.

The regularity — extensibility is traded against how small the base syntax is

Lay them out and something appears.

Language	Amount of base syntax	Ease of extension
Lisp / Racket	Minimal (essentially just parentheses)	Very easy
Forth	Essentially none	Very easy
Rust	Medium (constrained to token-tree shape)	Medium
Raku	Enormous	Hard — and they did it anyway

Lisp made syntax extension easy by having almost no syntax. A macro taking a list and returning a list suffices precisely because a list is the only representation a program has.

Raku is the opposite. Sigils, twigils, meta-operators, multiple dispatch, lazy lists — it kept an enormous syntax and made that extensible.

Here is this series’ thirteenth pattern.

How easy syntax extension is trades against how small the base syntax is.

And the price is actually being charged: the RakuAST note just above is exactly that. Lisp never has this problem. There is no grammar to break.

So where does Raku sit?

Being in tier 3 is shared with Racket, Seed7 and Forth. Two things are Raku’s own.

1. The grammar notation is a continuation of regular expressions. Writing Racket macros means learning a separate tool, syntax-parse. In Raku it is token / rule, and regex knowledge carries straight over
2. A grammar is a class, and it inherits. Inheriting an existing grammar and overriding a few rules is something you rarely see anywhere else

In other words, what is Raku’s own is not what it can do but the low entry cost and the shape of reuse. That is the sense in which this instalment’s title calls grammars “the thing only Raku has.”

Combined With the String Unit

As part 6 covered, Raku’s strings are grapheme-based. So . matches “one visible character.”

When you handle text with combining characters or emoji through regular expressions, the nuisance that other languages require simply never arises.

Part 4’s stance — buying correctness with implementation cost — pays off again here. Dedicated VM (part 6) → grapheme strings → correctness in regular expressions is one continuous line.

From Someone Building Their Own Language

I write the parser for my own language. Re-reading Raku’s grammars for this instalment, I took away two things.

1. The token / rule distinction is design, not implementation convenience.

“Does this backtrack?” and “does this skip whitespace?” are decisions a parser author makes every time. Making you write them as declarations converts an implicit decision into an explicit one. In my own parser those decisions are scattered as conventions about how functions are written, and they have no name.

2. One grammar, swappable Actions.

This has the same shape as the roast discussion in part 7. Write the same thing in two places and you get two different values. If the grammar definition is copied separately into the evaluator and the formatter, it will drift.

Raku offering this as a language feature means the room to drift has been removed structurally.

---

Next (final part): Raku in 2026. RakuAST becomes the default, and 6.e is coming. And how to write “small but ongoing” accurately.