Development Environment
A Mac on the desk, an Ubuntu VM beside it, and the homelab behind both.
Desk
What is on the Mac
macOS runs the editor and little else that matters. VS Code and a terminal both arrive over SSH on localhost:5555, which the container runtime forwards into the VM; the shell, the toolchains and Claude Code all live on the far side of that port.
Remote-SSH. The server side runs in the VM; the window on the Mac is a view of it.
ssh -p 5555 coder@127.0.0.1 — the same door, without the editor.
Port 22 inside, forwarded by the container runtime. Holds the door until /work answers, for up to 120 s.
Runs inside the sandbox, on the same shell. FLEET.md is imported into every session, so it knows the constraints.
Creates the VM from spawn, starts it again at each login, and builds the image from the tarball.
Sandbox
One VM, made from a tarball
An Apple container VM called dev: arm64 Ubuntu 26.04, six CPUs, 24 GB, and a rebuilt 6.18.15 kernel with netem and the shaping qdiscs compiled in. All of it comes from ../devcontainer — make packs the sources, the Mac builds the image, and spawn creates the VM with its volumes, binds, MTU and port.
A Dockerfile, a dozen scripts, supervisor.conf, the mise manifest, the kernel.
Checks every COPY is covered and packs one reproducible tarball.
Turns the tarball into the localdev image.
container run: CPUs, memory, five volumes, two binds, the kernel, port 5555.
No Docker daemon and none wanted; no qemu; /proc/sys read-only. Anything that needs another architecture goes to the fleet builder instead.
Boot
entrypoint, then supervisord
entrypoint.sh makes a fresh container safe to use — hands the empty volumes to coder, pins per-route MTUs, restores the metadata database if it is missing — and execs supervisord, which starts six programs by priority. Order is a preference, not a dependency; the one hard wait is sshd holding the door until /work answers.
Hands the empty volumes to coder, pins the route MTUs, restores the metadata DB if it is missing, then execs supervisord.
Mounts /work from the object store. Exits 78 and parks if the metadata is missing — no restart fixes that.
Streams the metadata SQLite to its own bucket. Parks with instructions if the credentials file is absent.
Converges /opt/tc against the manifest and exits. EXITED is its healthy state.
The sandbox’s native arm64 build node, half of the fleet builder.
Trims every ext4 volume at boot and daily, so the sparse images on the Mac shrink back. Skips a volume with the error flag set.
Last. Waits for the JuiceFS mount before accepting logins, so a reattaching VS Code window never opens on an empty tree.
Toolchains
A manifest in the image, payload on a volume
The image carries mise and its manifest; toolchains.sh installs what the manifest names onto the toolchains volume, so bumping Node is an edit plus a oneshot, not a rebuild. Single static binaries stay in the image. /etc/environment puts all of it on PATH for non-interactive sessions too, the VS Code server included.
Pinned in /etc/mise/config.toml. kubectl tracks the cluster, not the image.
Single binaries with no moving target to track.
Go, npm, pip, uv and everything XDG-conformant write to the devcache volume, never to the home directory.
Storage
Split by how you get it back
Five volumes, divided by recoverability rather than by content: identity worth snapshotting, toolchains rebuildable in minutes, caches that are pure churn. Projects live on JuiceFS at /work — data in the homelab’s object store, metadata in a SQLite file on a bind from the Mac, streamed to a bucket by litestream.
Identity and configuration only, small enough to snapshot.
One oneshot puts it back from the manifest.
Build and package caches. Never back this up.
A volume, not a bind: overlayfs needs a real Linux filesystem underneath.
Host fingerprints stay stable across image rebuilds.
~/code is a symlink to it. Metadata operations are slow here; never walk it recursively.
S3 on the homelab’s Ceph. Chunked, versioned, replicated by someone whose job that is.
Every WAL segment, a few seconds behind, on the same object store. Restored at boot if the file is gone.
Network & fleet
1500 on the link, 1280 over the tunnel
The link runs at 1500 so QUIC works; the ranges the Mac forwards over Tailscale are pinned to 1280 per route, and the homelab LAN to 1260. On the other end sit the fleet builder’s amd64 node, the cluster API and the object store — the homelab, one page over.
Link MTU 1500, so QUIC and HTTP/3 work everywhere that is not the tunnel.
Forwards the tailnet and the homelab LAN into the VM and clamps TCP MSS on the way, which is why only UDP ever noticed the MTU.
Three sites, two tailnets, one cluster.
buildkitd in this VM, over its socket. Never --load: there is no daemon to load into.
A buildkitd in the cluster, over the tailnet. Each architecture builds natively; nothing is emulated.
microk8s, reached over the tailnet. kubectl is pinned within one minor of it.
GHCR for the sites; the in-cluster registry over the service CIDR.