Bytes Get in the Door

The bitwise operators made a whole class of programs suddenly cheap to write, so the next probe was a CRC-32 checksum tool — thirty lines, the algorithm almost transcribable from its polynomial. It computed the correct checksum for the standard test string on the first try. The interesting failure was not in the arithmetic. It was in getting the file into the program at all.

The promise that held on one backend

Mere’s string has always been documented as bytes — the memory-model note says so, and the Unicode episode leaned on it. But “a string is bytes” is a claim that has to be true on every backend to mean anything, and it wasn’t. Reading a file returned a string, and on the C backend a string is a NUL-terminated char*. Hand it a binary file with a zero byte somewhere in the middle and it stops reading there, silently. The probe’s test file was twenty-five bytes with a NUL near the front; the C build read two of them and produced a checksum that was internally consistent, plausible, and completely wrong. The interpreter, whose strings are real OCaml strings that carry NULs without complaint, read all twenty-five and got the right answer. So the two backends disagreed — not with an error, but with two different confident numbers — and the one the tests ran on was the one that happened to be right. This is the same shape as the integer-width episode: a promise the language made was only kept where it was tested, and a binary file was the first input that reached past the place where it broke.

A vector of bytes, not a bytes type

The fix had a design fork in it. The honest binary-input path is a value that means the same thing everywhere, and the tempting move is to introduce a dedicated bytes type. The cheaper move — and the one taken — was to read the file into a vector of integers, one element per byte, each in the range 0 to 255, reusing the vector machinery the language already had. Eight bytes of storage per byte of file is an unflattering ratio, and it is written down as the honest cost; the day a program makes that ratio hurt is the day a real bytes type gets its forcing program. The interpreter and C backend got the reader; Wasm and LLVM reject it with a pointed compile error, because their host file interfaces still traffic in NUL-terminated strings. One subtlety earned its own debugging session: the new reader returns a vector, and the type variable naming the vector’s region has to be pinned at the point the reader is called — exactly as the vector constructor does it — or the region stays unresolved and the C backend silently declines to emit any function that takes the returned vector. The probe hit that too, which is fitting: an episode about a value silently failing to arrive found a second value silently failing to arrive, one level down in the compiler. The checksum tool now agrees with the system zlib on both text and NUL-bearing files. The next episode picks up a different file format entirely — and finds the documentation describing a language poorer than the one that exists.