I spent a full day on a “five-minute” Docker migration

We ditched Docker Desktop for Colima — and then PHP started segfaulting. A story about a simple migration that was not simple. On the way: a free tool that was not really free, a segfault with no logs, and a kernel that breaks its own contract. It ends deep inside the Linux syscall table — with a fix you can copy in five minutes.

The invoice that never came

It all started with a company decision. Docker Desktop is free for small companies. But when a company grows big enough, it must pay for every developer who uses it. Our company was getting close to that line. Nobody had paid anything yet — the invoice was still in the future. But it was clear that the invoice was coming, and that it would grow with every new hire.

So the decision was made early, before the first payment: we move to something free. It is much easier to migrate when you are not in a hurry. And honestly, the tool we would pay for is just a nice wrapper around free software. Paying for it felt strange.

Free for me, but I am a team player

Here is the funny part. This decision was not really about me. I work as a B2B contractor. My own little company is one person: me. By Docker’s own rules, a company this small can use Docker Desktop for free. Legally, I could keep it forever and pay nothing. Nobody would come after me.

But I did not want to be the special one. When one person in the team runs a different setup, small problems start. And this time, the problems would be mine, not theirs. The README would stop being true for me. New scripts would break on my machine only. Every little change in the stack would cost me an hour of debugging, alone. I did not want to be that person. So I decided to migrate like everyone else.

Podman: close, but no socket

The first idea was Podman. It is free, it is popular, and people say it is a drop-in replacement for Docker. So we tried it. It was not a drop-in replacement for us. Not even close.

The first problem was our SSH agent. Our docker-compose.yml mounts a special socket file into the PHP container: /run/host-services/ssh-auth.sock. Thanks to this socket, composer install inside the container can talk to our private GitLab over SSH. Here is the catch: this path is not a Docker feature. It is a Docker Desktop feature. Podman does not have it. Our composer.json has more than ten private repositories, so without the agent, nothing installs.

The second problem was host.docker.internal. Our PHP image has xdebug.client_host=host.docker.internal inside its config. Xdebug connects from the container to the host, where the debugger listens. With Podman, this name does not exist. Podman has its own name, host.containers.internal. The result: no debugging at all. Not in the browser, not for CLI commands. No error. Just silence.

The third problem was our Git worktrees. Every worktree has its own docker-compose.override.yml. These files use modern Compose features: top-level name: and the !override tag. The Python tool podman-compose cannot parse these files at all. It just crashes with a YAML error.

And there were small cuts too. For example, our lint script checks if the file /.dockerenv exists, to know if it runs inside a container. Podman does not create this file. It creates /run/.containerenv instead. Small thing. Easy to fix. But it was one more fix on the list.

Every single problem had a solution. But together they meant one thing: we would have to change shared files, shared scripts, and the habits of every developer. That is exactly the cost we wanted to avoid. So we looked somewhere else.

Colima: boring, and that is the point

Then we looked at Colima. Colima is different from Podman in one important way. Podman is its own engine that tries to act like Docker. Colima runs the real Docker engine inside a small Linux VM. It is not a reimplementation. It is the same dockerd that Docker Desktop runs.

This changed everything. The file /.dockerenv exists again, because real Docker creates it. The name host.docker.internal works out of the box, because Colima maps it by default. Modern Compose features work, because we use the real Compose with a real Docker socket. And best of all: Colima pretends to be Docker Desktop for the SSH agent. It creates the same socket path, /run/host-services/ssh-auth.sock, on purpose, for compatibility. You only need one setting: forwardAgent: true.

So the migration suddenly became boring. Uninstall Docker Desktop. Run colima start. Done. No changes in the repository. No changes in scripts. This is what we wanted. Well — almost. Keep reading.

Need for speed (enter Rosetta)

There was one more problem. Our main PHP image exists only for amd64. Our Macs have Apple Silicon chips. So the image must be translated. Colima’s default settings are safe but slow. The VM itself is fine — Apple’s own virtualization and fast file sharing are on by default. But Rosetta is not, so every amd64 binary crawls through QEMU’s emulation. The first boot was painful. Simple page loads took seconds. Sometimes more.

The fix was a few lines in the Colima profile config (~/.colima/default/colima.yaml):

vmType: vz          # Apple's Virtualization.framework (the default — keep it explicit)
mountType: virtiofs # fast file sharing (also the default)
rosetta: true       # the real fix: translate amd64 with Rosetta 2, not QEMU emulation
forwardAgent: true  # the SSH agent socket from chapter 3

With Rosetta on, the amd64 PHP-FPM container ran almost as fast as a native one. The worktree stacks came up. The tests ran. For a moment, the migration looked finished.

It was not finished.

The silent killer

After some real use, php-fpm started to crash. The container would start, live for one or two seconds, and die. The exit code was 139, which means SIGSEGV. There was no error message. No log line. No core dump. Just a dead container.

The strange part was this: some things worked, and some did not. The PHP CLI was fine: php -v, php -m, even our full unit test suite — over six thousand tests, all green. Only the FPM master process died. Always a second after start. Always silent.

The asymmetry has a boring explanation, and it is worth knowing because it sends you looking in the wrong place for an hour: opcache.enable_cli is 0 by default. The CLI never builds the shared memory segment at all, so it never touches the thing that kills FPM. Six thousand green tests told me nothing.

We made a small reproduction, to be sure it was not our app:

docker run --rm --platform linux/amd64 OUR_PHP_IMAGE php-fpm -t
# => exit code 139. Silence.

A clean image, one config test command, one crash. So the problem was below our code.

Same engine. Same Mac. Different crash.

Now here is the thing that made no sense. Docker Desktop and Colima both run the real dockerd. Both translate amd64 through the same Rosetta 2 on the same Mac. The exact same image worked on Docker Desktop for months. No crashes. Ever.

So if the engine is the same, and the translator is the same, then the difference must be somewhere lower. In the VM. Or in the way Rosetta is connected to the VM. This was the clue that later solved everything. But we did not know that yet.

Detective work

Time for the oldest debugging method: turn things off, one by one, until the problem goes away. It was the only method available, by the way — strace does not work under Rosetta, so there was no syscall trace to read. That is a large part of why this took a day. PHP makes bisection easy, though:

php-fpm -n -t                      # no php.ini at all      => OK
php-fpm -t -d opcache.enable=0     # all, but OPcache off   => OK
php-fpm -t                         # full config             => 139

So the problem was OPcache. But not the JIT part — our image already has opcache.jit=disable. The crash happens when OPcache creates its shared memory segment. OPcache maps this memory as executable, using the mmap syscall.

And then came the first real clue. We forced OPcache to use System V shared memory (shmget) instead of mmap. The crash was gone. Even on the completely stock PHP image, with no other changes. Same PHP code. Same Rosetta. Different kernel mechanism. So the problem was not “executable memory under Rosetta” in general. It was something specific about that one mmap call.

It worked on Docker Desktop. It should work here

At this point, I had a working workaround. I could stop. Ship the SHM change, go home, be happy.

But it bothered me. Docker Desktop runs the same Rosetta 2 on the same Mac, and it never crashed. Bugs in emulation are not random. When a low-level feature misbehaves, it is usually not an accident. Usually someone made a trade-off on purpose. Often for speed. I wanted to know who, and where, and why.

The monster had a name: MAP_HUGETLB

The answer was inside PHP itself — in OPcache’s shared memory code. When OPcache starts, it asks the kernel for a block of memory. If the size happens to be a multiple of 2 MB, it tries something special: an mmap call with the flags MAP_32BIT | MAP_HUGETLB | MAP_FIXED. It wants “huge pages” — bigger memory pages that are faster for the CPU. If the kernel says no, OPcache falls back to normal memory and everything is fine. This fallback is the key to the whole story.

On our Colima VM, the Linux kernel is built with huge page support (CONFIG_HUGETLBFS=y). So the kernel does not say no right away. It starts the operation, removes the old memory at that address — and only then fails, returning ENOMEM. A correct kernel never does this: a failed mmap must leave the old memory untouched. Now the process has a hole in its memory map. Under Rosetta, that hole is not empty — translated code was living there. The process touches the hole, and it dies. SIGSEGV, one second after start, no logs. This exact non-atomic mmap behaviour is described in the Colima issue we eventually found: abiosoft/colima#1452.

And why did Docker Desktop never crash? Here I have to be honest about what I verified and what I did not. I confirmed the Colima side directly — the crash, the flag, the fallback, the fix. For Docker Desktop I only observed that the same image never crashed in months of use; the explanation that its LinuxKit kernel rejects MAP_HUGETLB early, before touching the existing mapping, is my inference, not something I traced. It fits every observation I have, but treat it as the likely story rather than a proven one.

Same engine. Same Rosetta. Different kernel behaviour for one single flag.

One more data point that supports the mechanism: the crash depends on the size of the segment, not on huge pages being available. OPcache only attempts the huge-page mapping when the segment size is a multiple of 2 MB, so opcache.memory_consumption=65 sidesteps the whole thing by never asking. And enabling huge pages properly inside the VM does not fix it either — then php-fpm hangs instead of crashing.

Two rules to fix it all

Once you see it, the fix is elegant. We cannot rebuild the VM kernel. But we can make it answer “no” to huge-page mmap calls — the same clean “no” that Docker Desktop gives. Docker has a feature exactly for this: a seccomp profile, a list of rules saying which syscalls are allowed inside containers, and how.

So we took Docker’s default profile and put two rules on top of it:

• if mmap is called with the huge-page flag (flags & 0x40000) → return EPERM immediately,
• if mmap is called without it → allow, as usual.

Here they are, ready to paste:

{
  "names": ["mmap", "mmap2"],
  "action": "SCMP_ACT_ERRNO",
  "errnoRet": 1,
  "comment": "colima#1452: MAP_HUGETLB -> EPERM",
  "args": [{
    "index": 3,
    "value": 262144,
    "valueTwo": 262144,
    "op": "SCMP_CMP_MASKED_EQ"
  }]
},
{
  "names": ["mmap", "mmap2"],
  "action": "SCMP_ACT_ALLOW",
  "args": [{
    "index": 3,
    "value": 262144,
    "valueTwo": 0,
    "op": "SCMP_CMP_MASKED_EQ"
  }]
}

Two things are worth decoding here, because the OCI schema is not obvious. index: 3 is the fourth argument of mmap, which is flags. And for SCMP_CMP_MASKED_EQ the OCI spec uses value as the mask and valueTwo as the expected result — so the first rule reads “mask the flags with 0x40000; if what remains equals 0x40000, fail”, and the second reads “if it equals 0, allow”. 262144 is 0x40000 is MAP_HUGETLB. Rules are matched in order, first match wins, which is why the ERRNO rule has to come first.

Building the whole file is two commands — grab Docker’s default profile and prepend the rules:

curl -sL -o ~/.colima/seccomp-no-hugetlb.json \
  https://raw.githubusercontent.com/moby/moby/master/profiles/seccomp/default.json

jq '.syscalls = ([
  {"names":["mmap","mmap2"],"action":"SCMP_ACT_ERRNO","errnoRet":1,
   "args":[{"index":3,"value":262144,"valueTwo":262144,"op":"SCMP_CMP_MASKED_EQ"}]},
  {"names":["mmap","mmap2"],"action":"SCMP_ACT_ALLOW",
   "args":[{"index":3,"value":262144,"valueTwo":0,"op":"SCMP_CMP_MASKED_EQ"}]}
] + .syscalls)' ~/.colima/seccomp-no-hugetlb.json > /tmp/p.json \
  && mv /tmp/p.json ~/.colima/seccomp-no-hugetlb.json

Then tell Colima to use it as the default profile for every container. Two things in ~/.colima/default/colima.yaml:

docker:
  seccomp-profile: /etc/docker/seccomp-no-hugetlb.json
provision:
  - mode: system
    script: |
      cp /Users/<your-username>/.colima/seccomp-no-hugetlb.json /etc/docker/

That path has to be spelled out in full. The provision script runs as root inside the VM, where $HOME is /root — but your macOS home directory is mounted at the same path it has on the host, so /Users/you/... is what resolves. Substitute your own username.

One colima restart later, check it with the same command that started the whole investigation:

docker run --rm --platform linux/amd64 YOUR_PHP_IMAGE php-fpm -t
# => "configuration file /usr/local/etc/php-fpm.conf test is successful", exit 0

Everything works on the stock image. OPcache on. Default memory model. No ini files, no per-repo overrides. And because the fix lives in the VM and not in the repository, every new Git worktree works out of the box — no more silent crashes for the next person on the team.

One caveat before you copy this: the profile is the daemon default, so it applies to every container on that machine, not just PHP. Anything that genuinely wants huge pages — a database tuned for them, a JVM started with -XX:+UseLargePages — will now get a silent EPERM and quietly fall back. For a development machine that is a trade I will take without thinking. For anything resembling production, do not paste this without knowing what else runs there.

Notes for the next traveller

Most migration stories end with “and then it worked”. This one ended inside the syscall table. Someone else will hit this: they will see a silent exit 139 and they will have no idea where to start. So I left a comment with the root cause and the seccomp workaround in the GitHub thread that describes this bug: abiosoft/colima#1452. Maybe it saves someone a Friday afternoon.

If you just want the crash to stop and you do not care why, there are three smaller escape hatches, in rising order of how much they cost you:

• opcache.memory_consumption=65 — not a multiple of 2 MB, so OPcache never asks for huge pages. One line, keeps OPcache and the default memory model.
• opcache.preferred_memory_model=shm — switches to System V shared memory and avoids the mmap path entirely. This is the fallback I would use if the seccomp profile ever breaks.
• rosetta: false — back to QEMU. Nothing crashes, and everything is slow again.

What does not help, in case you are about to try: JIT settings, vm.overcommit_memory, ASLR, or enabling huge pages inside the VM. That last one is a trap — php-fpm stops crashing and starts hanging instead.

Was it worth it?

Honest answer: I do not know. The migration took me a full day, not the calm afternoon I hoped for. Maybe fixing our scripts for Podman would have been faster — those problems are annoying, but at least you can see them. A silent segfault inside a VM is a different kind of enemy. And the big question is still open: as a team, we have not decided yet. At first, Colima looked like the option with fewer problems. Then PHP started dying, and the fix was a custom seccomp profile. Both roads have holes. You only choose which holes you prefer.

Note, added later: I said “maybe Podman would have been faster.” I was wrong. I got curious and tested it myself. Podman does not use Rosetta by default — it is off, and being removed step by step. Without Rosetta, Podman uses QEMU instead, and that crashes too — but earlier and differently, with qemu-x86_64-static: QEMU internal SIGSEGV, before PHP even runs. This is a known, reported bug (containers/podman#22714), not something wrong with my machine. When I turned Rosetta on by hand, PHP did start, but then hit the same kind of crash as Colima — exit 139, same signal. My seccomp fix did not stop it either — this crash comes from a different syscall (munmap, not mmap with huge pages), later, when PHP shuts down. So Podman was not the easier road after all. Both of its emulation paths — QEMU and Rosetta — have their own version of this problem.

Minimal repro, both platforms:

Colima (needs rosetta: true in colima.yaml, see above):

colima start
docker run --rm --platform=linux/amd64 php:8.3-cli-bookworm \
  php -d opcache.enable_cli=1 -d opcache.enable=1 -d opcache.memory_consumption=128 -r 'echo "ok\n";'
# => exit 139, after printing "ok"

Podman (Rosetta is off by default, you must turn it on by hand):

mkdir -p ~/.config/containers
cat > ~/.config/containers/containers.conf <<'EOF'
[machine]
provider = "applehv"
rosetta = true
EOF
podman machine init --now

podman run --rm --platform=linux/amd64 php:8.3-cli-bookworm \
  php -d opcache.enable_cli=1 -d opcache.enable=1 -d opcache.memory_consumption=128 -r 'echo "ok\n";'
# => exit 139, after printing "ok"

Same flag (opcache.memory_consumption=128, a multiple of 2 MB). Same result. Two different paths to the same family of bug.

One more thing: I tested opcache.preferred_memory_model=shm on Podman under both emulation paths — Rosetta and QEMU. It fixed both. Clean runs, no crash, either way. So shm is not just a Colima trick — it works everywhere I tested it. If you want one fix that works regardless of platform, use shm, not the seccomp profile.

For now, my machine runs Colima, OPcache is on, and the Docker invoice will never come. The rest — the final team decision — is still an open question. Maybe that will be another article.