A Container Stack, All the Way Down: What It Took to Replace Colima in a Homemade Language

On a Mac, docker is a client talking to a daemon that is not on your machine. Something has to run a Linux kernel for it: Docker Desktop, or Colima, or Lima underneath them. I replaced that whole column — the virtual machine monitor, the engine, the runtime, the registry, the archive and compression formats — with programs written in Mere, a small ML-family language I am building.

The measure of done was fixed from the start and never moved: the real docker and docker compose, unmodified, have to be the client. Not a client I wrote. Not a compatibility shim. The binary Docker ships, pointed at a socket, getting the answers it expects.

It does that now. docker compose up brings up a two-service application inside the VM, the services find each other by name, a published port answers from macOS, docker exec -i pipes a script into a shell, docker push sends an image to a private registry and the real docker pulls it back out and runs it.

“I wrote Docker” is wrong in both directions

It overclaims. Docker is a CLI, BuildKit, containerd, Desktop, and an ecosystem. I did not write the CLI — and not writing it is the point. The client is the oracle. A client I wrote would only prove that my reading of the protocol agrees with my writing of it.

It also underclaims. Docker does not contain a VMM. On macOS something else supplies the Linux machine, and that is in here too — as is the language it is all written in.

The sentence that lands is: I set out to replace Colima, and wrote everything underneath it.

The stack, in published specifications

The interesting property of this project is not that it works. It is that almost every layer of it is a published specification, and there is exactly one vendor API in the whole column.

Layer	What I wrote	The specification	Scope, and what is deliberately absent
Client	(nothing)	—	The real docker / docker compose is the customer and the oracle
Engine API	mengd	Docker Engine API (announces v1.54, min 1.40); HTTP/1.1 with chunked bodies and Connection: Upgrade hijacking	images, containers, exec, networks, volumes, build, events, auth. No BuildKit — the classic builder only
Images	mengd + mtar	OCI Image Spec v1.1.1 (vnd.oci.image.{index,config,layer}) and Docker manifest v2; the whiteout convention (.wh., .wh..wh..opq)	Manifests, configs, layers and whiteouts in both directions. The manifest type follows the layers
Registry	mreg (server) + mengd (client)	OCI Distribution Spec v1.1.1, bearer-token auth, Basic (RFC 7617), TLS	Pull and push. Blob upload is the two-step form
Runtime	mrun	OCI Runtime Spec (config.json)	namespaces (including joining one by path), mounts, pivot_root, process/env/cwd, capabilities, masked and read-only paths, cgroupsPath and resources. No hooks, no seccomp, no LSM, no uid mapping
OS facilities	—	Linux: cgroup v2, overlayfs, namespaces, rtnetlink, bridge ioctls, RFC 4193 unique-local addresses	Used, not implemented. Claiming otherwise would be a lie
Devices	mvm	VIRTIO v1.3 (mmio transport, VIRTIO_F_VERSION_1 only, block with one queue, vsock with three)	Indirect descriptors are not offered, and neither is vsock SEQPACKET — what you do not offer, a driver cannot ask for
Boot	mvm + mkdtb	Booting AArch64 Linux, the Devicetree Specification (generated, not compiled), ARM PSCI	GIC, PL011, PL031, and the two PSCI calls that end a machine
CPU	mvm	Apple Hypervisor.framework — the only vendor API in the column	—
Formats	mtar / mgz	POSIX ustar (IEEE 1003.1); RFC 1951 DEFLATE and RFC 1952 gzip	tar reads and writes; PAX, GNU long names and base-256 sizes are refused by name. gzip goes both ways — the compressor is mine too

Two things in that table are not open standards: Apple’s Hypervisor.framework, and the Docker Engine API. The second is Docker’s own, published and versioned — and note where it sits. I am not implementing a client for it. I am answering a real one.

What it is made of

Component	Equivalent	Mere	C shim	Checks
mvm	the VM part of Lima/Desktop	1,945	909	1,513 lines, 10 gates
mengd	dockerd	4,933	1,502	1,256 lines, 5 gates
mrun	runc	767	390	a runc oracle
mreg	distribution	634	138	—
mtar / mgz	tar / gzip	1,739	254	962 lines, 11 gates

The C is only what the FFI boundary demands: sockets, netlink, ioctls, setns, waitpid, OpenSSL. Every algorithm — inflate, deflate, tar, HTTP, JSON, the virtio rings, the vsock state machine — is in Mere.

Four decisions that shaped it

The oracle is always somebody else’s program. mrun is compared against runc field by field: the same bundle handed to both, 130 fields across five cases, differences reported individually rather than as a pass or a fail. The layer writer is checked against the kernel itself — mount a real overlay, run a step, and require that merged == apply(lower) then apply-as-layer(upper). The push path is checked by having the real docker pull back what mine sent. None of these ask my code whether my code is right.

A layer is the difference a step made, and I cannot compute a difference — but the kernel can. Build steps run on an overlay mount, and the upper directory is the layer. Nothing had to be invented to store a build cache either: a step that has run before is one whose upper directory is already sitting there. The thing was already on disk under a name nobody had thought to use.

Configure the namespace before the process exists. There are two ways to give a container a network: let the runtime make a namespace and wire it up afterwards, or make it first and hand it over. The first is a race the container can lose — and will usually win, which is worse, because a failure that is rare gets blamed on the network. The OCI spec allows a namespace to be named by path, which is how CNI does it, so the wiring is finished before anything can look at it.

NAT is not missing; it is meaningless here. The guest has no network interface at all — the VMM gives it a block device and a vsock device, and everything in and out goes through the latter. NAT means translating onto an upstream, and there is no upstream. Writing that down, with the three ways to change it and what each would cost, is more useful than a half-built version carrying the name.

Numbers

	mvm + mengd	Colima
docker run --rm alpine echo	418 / 432 ms	428 ms
docker ps	90 ms	105 ms
guest boot to daemon listening	1.2 s	—
vCPUs	1	6

One vCPU, and the ordinary operations are at parity. Elsewhere: one layer per build step took the last step’s layer from 8,940,544 bytes to 2,048 and then to 116 once it was compressed; the build cache took a two-step build from four seconds to zero; my gzip is 0.6% larger than the system gzip on a real image layer, at 1.3 seconds where my first attempt took 369.

The bugs that only real use could find

Every one of these passed every check I had at the time.

The compressor that compressed nothing. mgz had three checks and all of them asked whether the output was correct: gunzip accepts it, the CRC agrees, the bytes come back. A compressor that emits stored blocks passes all three. Pointed at a real image layer, it produced 10,023 bytes from 10,000 — bigger than the input — because one code-length code wanted eight bits and the whole block fell back to stored. The missing check compares size against gzip.

A key is not a substring. docker compose build sends both t= and target= in its query string, and my parameter lookup found the t= inside target=. The image came out untagged; the build said “Successfully built” with no “Successfully tagged” after it, and compose then failed with “No such image” about the image it had just built. Nothing in the build was wrong.

A daemon that looks dead because it is full. Mere’s spawn returns a handle, and throwing it away leaves a joinable thread behind — the runtime says so, in the source, where detach is defined. I threw one away per connection. Add a /wait that polled for five minutes after the client had gone, and at about thirty containers the runtime would not make another thread and the daemon stopped accepting anything: compose produced no output and the trace showed not one request arriving. Forty containers went from 304 seconds to 14.

Killing a parent orphans a child. The runtime is the parent; the container’s init is its child. docker stop signalled the runtime, and docker rm -f removed the directory — both left the container’s processes running with nobody who knew about them, one per removal, until the machine had more runtimes than containers.

A store write cannot be allowed to raise. In Mere a raise ends the process. write_file into a directory another request is removing ends the daemon, not the write. It died mid-removal with a path as its last word — twice, the second time straight through a file_exists guard that the race stepped over.

And once, the client was the broken thing. compose up --build stopped working, and the daemon was innocent: on this macOS Docker CLI, DOCKER_BUILDKIT=0 docker build hangs forever — against the real docker too. The hang looked exactly like a daemon that had stopped answering. What proved it was not was pointing the same command at a daemon nobody doubts. The gate asks that question first now, with a deadline, and skips by name: a check that cannot tell “the daemon is broken” from “the client is broken” would hang, and a hang says nothing at all.

What it does not do

Said in the vocabulary of the specifications, because a boundary you can name is a design decision and a boundary you cannot is a gap:

• Outbound NAT — no upstream interface exists to translate onto
• OCI hooks, seccomp, LSM labels, uid mapping — not read from the bundle
• VIRTIO indirect descriptors, vsock SEQPACKET — not offered
• PAX, GNU long names, base-256 sizes — refused by name rather than guessed
• BuildKit, swarm, plugins, TTY — docker exec -t wants a pty in the container, which is a different mechanism again

Why do this at all

It is dogfood. The question was never “can I have my own Docker”; it was whether a language I am designing can carry a real system. The most valuable output is not the stack — it is the list of defects it found in the language: that region reclamation is lexical, so wrapping a call in a region does not reclaim what the callee allocated; that the C backend’s lambda lifter resolves free variables by name, so an inner closure breaks when the importing program happens to bind the same name; that a discarded thread handle leaks; that the arena “pointer” a program holds is an offset, so a shim that treats it as an address writes from somewhere else entirely and reports success.

None of those would have shown up in a test suite. They showed up because something real was asked of it, by somebody else’s client, with somebody else’s program as the judge.