Homelab Architecture
Currently three sites, two tailnets, one cluster.
Applications
The things with users: public sites, a set of MCP servers that hand Claude real data, and a bot. Each is a small container on the cluster, reached by hostname through the edge.
Public site, served through the edge.
This site. Built into an image on every push, deployed from git.
BMW ISTA service data, handed to Claude as tools.
Barry’s Bootcamp schedules and bookings, as tools.
ExpertFlyer seat availability and flight status, as tools.
Motorsport events from MotorsportReg and Speed District, as tools.
Small utility that answers my apartment building callbox and allows entering of access codes for door unlocking.
Media services
The library and its machinery
Viewers and the prober reach Plex through an edge cache, nearer to wherever they are. Around it: Seerr to ask for things, Tdarr to normalise them, and an exporter feeding the observability layer. The one workload with an audience that notices.
Keeps checking that playback actually works, the same way a viewer would.
Intercedes on the end-users behalf to build a more localized buffer of media chunks during a playback session. Applies TCP tuning over high latency paths to maximize throughput.
The library, and every stream out of it.
Ask for a film or a series; the rest follows.
Normalises the library in the background, across the fleet.
Exports what Plex is doing to the observability layer.
Observability & management
Watching and reaching all of it
Alloy ships metrics, traces and logs from every layer to Prometheus, Tempo and Loki; Pulumi declares the infrastructure below; and when the uplink or the rack is dark, a cellular link and a KVM still reach the metal.
Metrics, traces and logs from every layer, collected by Alloy and kept in one place.
Declares the cluster, the tailnets, DNS and the edge, so all of it is rebuildable from a repo.
A way in that depends on neither the uplink nor the rack’s own network.
Console into any server through a serial-controlled KVM switch, for the day a node will not boot.
Foundational services
What everything above stands on
Platform pieces the layers above take for granted: VMs and microVMs when a container is not enough, S3 on Ceph, a distributed database, a remote builder, and the operator that puts all of it on the tailnet.
Virtual machines as cluster objects, with their disks on Ceph.
Kata Containers on Firecracker: pods with a VM boundary, for anything that shouldn’t share a kernel.
S3 on Ceph. Used for any scenario where S3 is needed.
Puts in-cluster services on the tailnet without opening a port.
Remote builder for the fleet; amd64 images are built natively here.
Distributed SQL for anything above that needs a database.
Network
Two tailnets, one doorway
Two tailnets keep people and infrastructure apart. The Traefik edge faces the public internet and the users tailnet through the host’s kernel Tailscale, terminates TLS with certificates it fetches from ConfigFS, and forwards each hostname over the services tailnet — the backend — through a tsnet connector built into Traefik.
Terminates TLS and routes by hostname. The only way in.
tsnetWhere everything talks to everything. No people on it.
Routes and TLS certificates.
Logical
One cluster, one storage system
All nine machines join one cluster with a single control plane, at home. Rook runs Ceph across the rack’s seven nodes only; everything stateful — volumes, VM disks, buckets — lands there, and the Fly.io nodes hold none of it.
Physical
A rack at home, Fly.io abroad
Seven servers in a rack at home on a 10 Gbps network, five of them full of disks, and a Fly.io machine each in San Jose and Sydney. The rack reaches them over the public internet on a symmetric 5 Gbps uplink; everything above this line is indifferent to which side of it a workload lands on.
Bare-metal servers on a 10 Gbps switched network.
A Fly.io machine on the US west coast.
A Fly.io machine on the Australian east coast.