Freed, Not Returned: A Blowup in Mere's Runtime, mmap on macOS, and a Fixed Threshold on Linux

Mere is a small language I build for myself. The runtime of its C backend manages memory in regions (arenas): a region takes blocks from malloc, chains them, and frees them all when the program leaves the region. There is no garbage collector. Small memory and a known moment of release are among the things this language is for.

On top of it sits mere-ruby, a Ruby implementation written in Mere. It has its own collector for Ruby values, and its Fibers are Mere coroutines.

This is about one mere-ruby test that went past 6 GB and kept going. The causes were a quadratic walk in the runtime, and malloc. At first the malloc problem looked like macOS’s alone; later Linux turned out to have it too, and each OS got its own fix. The numbers come from macOS 26.7.1 on Apple Silicon (32 GB of RAM) and from an Ubuntu 24.04 container (glibc 2.39, kernel 6.8, arm64).

The table first

	Before	After
The test (thousands of suspended Fibers, a collection every 50 safepoints)	Could not get through its first loop in 300 s; past 6 GB once that was fixed	Finishes in 181 s, peak 3.7 GB
Peak RSS of the same program, run normally	1.08 GB	0.74 GB (−31%)
CPU time of mere-ruby’s 472 corpus programs	75.3–77.8 s	75.7–79.1 s (within the noise)
Sum of their peak RSS	79.7 GB	78.4 GB (−1.6%)
ruby’s test_kwarg_eval_memory_leak (growth limit 1.2)	Fails at 1.209	Passes
RSS right after leaving a 400 MB region	841 MB	1 MB
Linux: the same test	Past 6 GB in 17 s	Finishes in 244 s, peak 3.6 GB
Linux: peak RSS, run normally	788 MB	598 MB (−24%)
RSS right after leaving a region of 8,000 compacted maps (macOS)	991 MB	42 MB

The macOS test numbers under “After” are with the malloc fix (v0.1.686) alone. With the later map fix (v0.1.688) the peak drops to 2.5 GB.

A test that would not finish

mere-ruby has a mode for shaking holes out of its collector: MERE_RUBY_GC_STRESS=50 runs a collection at every 50th safepoint. One of the 472 corpus programs did not finish under it. It creates an Enumerator, steps it once and drops it, thousands of times. Each Enumerator holds a Fiber – a suspended coroutine.

sample put 95% of the time in one runtime function: the one that, before a region is retired, asks whether any suspended coroutine’s stack points into it. If one does, the region is kept; freeing it would let the coroutine read freed memory when it resumes.

That was quadratic. Every retirement tried every retired region again, and every try walked every suspended stack. With thousands of suspended Fibers, that is slow.

A suspended stack does not change until its coroutine runs. So a retired region now records when it was last tried, and while none of the coroutines that pinned it has run since, it stays retired without the walk. The only work skipped is work whose answer can only be “still pinned”, so no region is freed any later than before. This fix has nothing to do with the OS; it is v0.1.685. The 3000-iteration loop went from over 300 s to 46 s.

The test still did not finish. In its second half it passed 6 GB, and the guard (a script that kills a process past an RSS cap) killed it.

Two wrong guesses

The first guess was the number of Fibers. Within one statement, suspended Fibers are reclaimed once 1024 of them are live. Maybe under the stress mode that failed, and they piled up by doubling. A print showed the reclaim working every time: 1024 down to 85. Wrong.

The second guess was the retired regions. Maybe suspended Fibers kept pinning old regions, and those piled up. Printing their count and size at every retirement showed tens of megabytes at most. Wrong again.

vmmap showed what the memory actually was, on the way to 6 GB:

MALLOC_SMALL           resident  446 MB
MALLOC_SMALL (empty)   resident  2.6 GB
MALLOC_LARGE           resident  650 MB
VM_ALLOCATE            resident  112 MB   <- the Fibers' stacks

The largest line was MALLOC_SMALL (empty): memory already freed, which malloc was holding on to. The runtime was freeing its regions. What did not come back was everything past that.

macOS does not give it back

A small C program: allocate blocks of one size, write all of them, free all of them, and read the numbers right after.

macOS 26.7.1, right after free

Allocated and freed	Category	RSS	phys_footprint
64 KB × 1000	MALLOC_SMALL	still 64 MB	still 64 MB
1 MB × 200	MALLOC_SMALL	still 202 MB	still 202 MB
16 MB × 12	MALLOC_LARGE	still 193 MB	still 193 MB
64 KB × 1000 with mmap/munmap	—	1 MB	1 MB

At no size did free alone return a single byte. phys_footprint – the number in Activity Monitor’s Memory column, and the one memory pressure is judged by – did not move either. malloc_zone_pressure_relief(NULL, 0), which tells malloc the process is under pressure and should give back what it can, answered 0 bytes.

The default allocator in current macOS is xzone malloc, and it appears to keep freed blocks at hand for the next allocation. Another project reports the same on macOS 26.3.1: blocks from 16 KB to 4 MB stay in the footprint, and pressure relief returns 0.

Linux (glibc 2.39), right after free

Allocated and freed	RSS right after free	After malloc_trim(0)
64 KB × 1000	still 63 MB	1 MB
256 KB × 400	1 MB	1 MB
1 MB × 200	1 MB	1 MB
20 KB × 4000, every other one freed	77 MB	47 MB

glibc serves anything from 128 KiB up (a threshold that then rises dynamically, to at most 32 MiB) with mmap from the start, so free returns it at once. Smaller blocks stay in the heap, but malloc_trim(0) gives back every whole free page, even from the middle of it.

The two policies seemed to point opposite ways: glibc returns large blocks at once, while macOS keeps them to reuse. Mere’s regions allocate and free in large units, over and over, and everything macOS kept added up – or so I thought; at this point I took it for a macOS problem.

The fix: map the blocks of 16 KB or more

On macOS only, a region block of 16 KB or more is now taken with mmap instead of malloc, and given back with munmap when freed. Smaller blocks still come from malloc.

• Why 16 KB: it is the page size on Apple Silicon. The smallest mapping is one page, so mapping smaller blocks one at a time would round each up to 16 KB.
• Why macOS only, at the time: two reasons (the first turned out to be wrong).
  • glibc already maps large allocations and returns them, so there is little to gain on Linux.
  • Linux limits the mappings a process may have with vm.max_map_count. The kernel default is 65,530, and a mapping per block could reach it. Ubuntu 24.04 and Fedora 39 raised it to 1,048,576, but many machines still run the old default.
• Growing a block: macOS has no mremap, so the realloc that grows a block maps a new one, copies, and unmaps the old one.

I also tried keeping returned mappings at hand, up to 32 MB, and handing them to the next allocation of the same size. It was slower: only exact sizes could be reused, and looking for one cost more than it saved. It did not go in.

Does this cost Mere what it is for?

The question I was asked was whether this change gives up what makes Mere worth using. I measured.

• Nothing is freed later. A region still gives its blocks back when the program leaves it; they now go to the OS instead of to malloc’s hand.
• Speed is about the same. The CPU time of the 472 corpus programs moved within the run-to-run noise (about 3%). On the one program that allocates and frees hardest, the system calls show: one run went from 2.88 to 3.35 s – though the same binary ranges from 2.81 to 3.26 s on its own.
• Memory goes down: a little in ordinary programs, a lot in the ones that blew up.

The gate that checks it grows a region to 400 MB, leaves it, and reads the resident size from outside with ps: it must be under 64 MiB. The new runtime holds 1 MB there; with the poison that puts malloc back, it holds 841 MB and the gate goes red.

I measured the wrong number

At one point I reached a wrong conclusion.

malloc can also be tuned with environment variables. I ran the test with MallocSpaceEfficient=1 and a few others, saw the peak RSS unchanged, and concluded that no setting gives the memory back. For a while Mere’s changelog said so.

Measuring again for this article showed otherwise.

Right after free	64 KB × 1000	1 MB × 200	16 MB × 12
Default	RSS 64 / footprint 64 MB	202 / 202 MB	194 / 193 MB
MallocSpaceEfficient=1	RSS 64 / footprint 1 MB	202 / 2 MB	1 / 1 MB
MallocLargeCache=0	RSS 64 / footprint 1 MB	202 / 2 MB	2 / 1 MB

With these settings, freed memory is marked reusable. It leaves the footprint at once, and stays in the resident size until the kernel actually takes it. I had looked only at RSS, and only at its peak – and a peak says nothing about whether freed memory came back, because whatever was built up before the free stays the peak.

The decision stood: with the default settings, neither number goes down, and an environment variable is set when a process starts, not something a runtime can reliably change from inside. But the changelog was corrected before it shipped.

The second mistake: Linux grew too

The first version of this article listed, under what was left, that I had not run the test on Linux. The next day I did. On Linux the same test passed 6 GB in 17 seconds.

glibc’s mmap threshold is dynamic. It starts by mapping anything from 128 KiB up, but once a mapped block is freed, it raises the threshold to that block’s size, and from then on allocations up to that size come from the heap, where a freed block stays. The small C program above allocated and freed only once, so it saw only the first round. Mere’s regions allocate and free blocks of much the same sizes again and again.

Linux	Peak RSS, run normally	The test
Default	788 MB	Past 6 GB in 17 s
malloc_trim(0) every 64 MB freed	608 MB	Past 6 GB in 105 s
Threshold fixed at 64 KiB	574 MB	Finishes in 207 s, peak 3.6 GB
Mapping blocks of 16 KB or more ourselves	555–564 MB	Finishes in 258 s, peak 3.6 GB

An occasional malloc_trim could not keep up. In v0.1.688, on glibc a program fixes the threshold when it starts, with mallopt(M_MMAP_THRESHOLD, 64 KiB) – unless MALLOC_MMAP_THRESHOLD_ is already set, and not in a library build, since a library should not change a process-wide setting behind its host’s back. It is faster than mapping blocks ourselves, and when a mapping fails or the count limit is reached, glibc falls back to the heap on its own, so the mapping limit is not a worry.

The gate now answers on Linux too: 5 MB resident right after the region with the fixed threshold, 405 MB with the poison that leaves it dynamic.

The same version fixes another leak found along the way. A map or a Vec gets an arena of its own when it is compacted. That arena was not freed when the program left the region the map was made in: the map’s struct went with the region, and its arena went with nothing. The arena is now tied to the region its struct lives in and retired with it. Right after leaving a region of 8,000 maps, the resident size on macOS went from 991 MB to 42 MB.

What is left

• Blocks under 16 KB are still kept by malloc. A program with many small regions can still grow.
• A mapping per block costs a system call. It does not show in ordinary programs, but it does when allocation churns hard. I have not yet found a way to reuse blocks that costs less than it saves.
• The test finishes now, but peaks at 3.7 GB. Thousands of suspended Fibers and a collection every 50 safepoints are heavy to begin with.
• On Linux the threshold is process-wide. A C library a Mere program calls through the FFI also gets its allocations of 64 KiB and up mapped. A library build leaves the threshold alone.

What I would tell myself before starting

• If the runtime does free, suspect malloc next. Between free and “returned to the OS” sits the allocator, and its policy differs by OS.
• Peak RSS does not tell you whether memory came back. Read the numbers right after the free – on macOS, both RSS and footprint.
• Measure your way out of a guess. The Fiber count and the retired regions each fell to a single print. One line of vmmap was the answer.
• macOS numbers change between versions. The allocator itself has been replaced. Measure on your own machine before deciding.
• Do not judge an allocator by one allocation and one free. glibc gave memory back the first time and kept it from the second on. Measure the shape of the real program, repeated.
• Put a guard on it. The day before this investigation, an unguarded test grew to 43 GB; macOS held on with compression and swap until a watchdog kernel panic stopped the machine. On Linux, the OOM killer would have killed that one process.

The fix for the quadratic walk is in Mere v0.1.685, the macOS mapping in v0.1.686, and the Linux threshold and the map arenas in v0.1.688. Each comes with a gate in Mere’s repository, and with a poison that turns the gate red when the fix is taken out.