Plex Edge Cache, Prober and Performance Monitoring

Architecture

The edge cache

A player that asks for 256KiB, waits, plays it, asks for the next 256KiB and waits again pays a full round trip per request. At 300ms that is 300ms of dead air for every quarter-megabyte, which caps the stream at well under a megabyte a second no matter how much bandwidth exists between the two ends. The player’s own buffer is small, so the first stall the network produces is the one the viewer sees. Well-written clients hide this by keeping several requests in flight and buffering minutes ahead. The Plex client ecosystem is not uniformly well written, and smart-TV apps in particular are constrained, old, and not something I can fix.

So the fix goes where it can be applied: a process beside the Traefik edge node that reads ahead of wherever the client is, fetches in parallel so the pipe is actually full, keeps stream state between requests so a hundred small ranges are one stream being run ahead of rather than a hundred cold starts, and shares the result so the second viewer of a file reads it locally. Nothing is required of the client. It works the same for a Tizen app as for the desktop player, which is the point.

The read-ahead is measured rather than configured. Nothing in a range request states the media’s bitrate, but a player’s long-run consumption rate converges on it, since that is the only rate playback can sustain. The cache tracks bytes delivered over a sliding window and keeps a target number of seconds ahead of the read position, clamped between a floor and a ceiling. A 3Mbps phone stream with a 60 second target gets about 22MiB of read-ahead; a 90Mbps remux gets the ceiling; a stream that has just started has no rate yet, so it gets the floor, which exists to give the opening a head start.

That is the design. What follows is the part that only shows up once real players are on it: the special cases, each of which was a measured problem before it was a feature.

Not charging the viewer for the disk

A fly volume is rate-limited per machine, measured at 17.8MB/s on these nodes, well below what the link delivers. Gating delivery on the write made the cache charge a viewer for its own bookkeeping: on a miss, a 26ms first byte became 224ms, against 26ms for the same read proxied straight through. The cache was adding latency on exactly the path it exists to improve. Each in-flight chunk is now buffered in memory and a reader is served from that buffer as the bytes arrive, with the write to the store happening behind them.

Starting the fill where the reader is

A chunk is the unit the cache fetches in, but a client’s range rarely begins on a chunk boundary; on average it begins half a chunk in. A fill that started at the boundary made the client wait for every byte between it and their offset, about 2MiB with 4MiB chunks, on the very request that decides how long pressing play takes. A fill a reader is waiting on now begins at their offset and runs to the end of the chunk, and the head is fetched afterwards so the chunk still completes for the store. That is one origin request more per cold miss, in exchange for a first byte that arrives one round trip after asking.

The pause, the seek, and the viewer who left

A paused client’s rate decays, its window shrinks, and the cache stops spending origin bandwidth on somebody who is not watching. When they resume, the estimator restarts rather than averaging the resumed traffic over the idle time, so the buffer comes back promptly. A seek abandons the in-flight fetches and re-anchors the window, so a viewer who scrubs repeatedly gets no benefit, and that shows up as exactly that rather than as a mystery. The player’s own timeline reports pass through the same node, so the cache reads them on the way past without touching them: a stall is the cache’s guess that a viewer suffered, a rebuffer is the player saying so, and a stall while the player was paused is read-ahead nobody watched.

Eviction that is not an LRU

A stream’s already-played chunks are, by definition, touched more recently than the chunks fetched ahead of it, so a plain LRU would throw away exactly the buffer the cache exists to maintain. Chunks are held in three tiers and given up in order: idle, files nobody is watching; spent, bytes an active viewer has already played; and ahead, read-ahead a viewer has not reached. The third is not really eviction. When the only bytes left belong to somebody’s buffer, a speculative fetch is refused and that stream reads less far ahead. A fetch a viewer is actually waiting on is never refused. Separately, one large title cannot quietly become the whole store: a per-title ceiling defaults to a quarter of it, and what a file gives up when it binds is whatever is furthest from where it is being read.

Transcodes are a different thing

A transcoding client never touches the original file. It reads numbered segments from a session the server deletes the moment playback stops, so there is no bandwidth to save and the offload ratio is zero by construction. What there is to save is the round trip on the next segment, which is the whole point when the client is a television with a small buffer. Because the position is the number at the end of the URL, asking for segment n opens the ones after it in the background. Every segment fetched ahead makes the transcoder produce that much further ahead, so this buys buffer with the origin’s CPU, and it defaults to two for that reason. The subtle part is a 404, which means one of two opposite things: the end of the media, where everything held is still good, or a session torn down, where none of it is. The cache tells them apart by asking after the session’s initialisation segment, which lives as long as the session does. A session served entirely from hits would never find out it had ended, so it is asked after on a timer anyway.

The index at both ends of the file

A client reads the head and the tail of an MP4 at every open, because the index lives at one end or the other, and it does so after every pause, scrub and reconnect, whatever it is about to play. Measured on an iPhone session, those two reads came from memory at 0.3ms while the chunks were resident and from the bucket at about 500ms each once they had aged out. A rebuffer mid-episode turned out to be exactly that: a re-open paying two bucket round trips for the index before any media byte could flow. So a title’s first and last chunks are pinned while it has a viewer and for thirty minutes after the last one leaves, because the pauses people actually came back from ran five to twenty-five minutes.

Living in a bucket

A fly machine’s volume is small and slow, so the store can live in an S3-compatible bucket or in GCS instead, and a bucket outlives the machine: a redeployed node comes back to everything it held. That changes the shape of the problem. A hit that is no longer in memory is a round trip to the bucket, so the read-ahead warms complete chunks back into memory while the viewer is still a window behind them, and the number of concurrent warms is bounded, after unbounded ones turned a few streams’ windows into dozens of slow fetches sharing one link. Deletes are batched in the background, and to make that safe nothing is ever written under a name that has been used before: every fill gets a fresh nonce and every admission of a title a fresh generation, so a delete in flight can only ever remove the copy it was issued for.

Where the bucket is matters as much as what it is. From the Sydney node, Tigris kept its bytes in Sydney but answered every lookup from elsewhere in the region, a constant 105ms on top of the transfer, while a regional GCS bucket in the same city answered in 20ms and streamed a chunk at nearly twice the rate. GCS’s own DNS-balanced endpoint then landed the connection on front ends 100ms or 230ms away, re-rolled on every reconnect, until it was pinned to the bucket’s regional endpoint.

Taking a device out

Every stream is read ahead by default, and the one thing you can be sure of with a fleet of client platforms is that one of them will behave badly. Exclusions are an ordered rule list over what a request carries: client id, product, platform, model, user, remote address and so on, with globs. The policy file is reloaded in place when it changes, because taking a misbehaving television out of the read-ahead path should not need a redeploy of an edge node in another hemisphere. A rule can also hand a class of device a longer buffer than the default, which is what the smart-TV apps get. And a single request can choose its own caching with a secret token, which is what lets the prober measure the cache against its own absence.

What it adds up to

The prober’s three-way read is the honest measure of all of this, and the vantage point that matters is Sydney: 155 ms from the origin, which is the kind of distance the edge exists for. The direct route over the tailnet, kernel WireGuard from the same machine, medians at 95 Mbps against a 102 Mbps title. It cannot keep up. The edge with the cache told to stand aside reaches 218 Mbps, and that bar is the Flyscale transport alone: BBR with fq pacing and no slow start after idle, terminating the viewer’s connection a few milliseconds away and talking to home over one tuned path, with nothing cached. A cold cache reaches 296 Mbps because the fill runs ahead in parallel, but it pays for the fill before anything flows, and it shows: 603 ms to the first byte, the worst of the four. A warm cache reaches 698 Mbps, nearly seven times what the title needs, with a first byte in 24 ms, because the bytes were already in Sydney. HTTP/2 does better still, 839 Mbps warm.

Unfortunately there is no before and after for the TCP tuning: BBR has been in the edge image for longer than Prometheus remembers, so the bypass bar is the edge transport as a whole, tuning included, against a plain kernel path from the same machine. And from San Jose, 7 ms from the origin, the direct route wins at 695 Mbps and no version of the edge beats it. A probe close to the homelab will always favour going direct. The edge is for viewers who are far away, and the chart is what far away looks like.

The prober

The prober is a small Go process that runs from wherever I want to know how Plex feels from, one per region I care about. On a timer it walks a matrix of route, address family and protocol, and on every cell it does two things: a ladder of small requests, then an authenticated ranged read of a real media file.

The two routes are the point. The edge route goes to the public hostname, which anycast DNS resolves to the nearest fly.io node, through Traefik and the cache, and home over the tailnet. The direct route dials the origin from a Tailscale node embedded in the prober itself with tsnet. Both are confirmed to be the same server by comparing the machine identifier each returns, and the difference between them is exactly what the edge tier is costing or saving from that vantage point.

The ladder is built so a regression can be attributed rather than just noticed. /identity needs nothing; /status/sessions adds the token; /library/sections adds a database query; /web/index.html adds a static file of about 30KB. Each step adds one thing to the one before. They are milliseconds of work, so their timings are nearly pure latency, and they run before the media read so a connectivity baseline still lands if streaming later times out.

The stream read is the question the tool exists to answer: not “is Plex up” but “can this location pull media fast enough to play it”. It picks the highest-bitrate item in the library by default, so the answer is for the worst case. The timing is split into connect, TLS, time to first byte and backhaul, which is what is left after the handshakes are subtracted: on the edge route that is the node reaching the homelab plus the origin’s think time, on the direct route the origin’s think time alone. Splitting connect from backhaul separates “the nearest edge node is far from this viewer” from “the backhaul to the homelab is slow”, and those have completely different fixes. For the read itself, three numbers: the mean throughput, the throughput floor, which is the slowest sampling window after warm-up because a mean hides a mid-transfer collapse, and the realtime factor, goodput divided by the title’s bitrate. Above 1 it streams without buffering. That is the number I alert on.

A measurement is only as good as its method, so a few rules are enforced rather than assumed:

• Legs run one at a time, never in parallel, so they cannot end up measuring each other.
• Every request gets a fresh connection, because a media player pays the handshake too.
• Read offsets are randomised. A fixed offset would be served from the origin’s page cache after the first cycle, measuring the homelab’s RAM rather than the path to it.
• The negotiated protocol is verified on every response, so a leg that silently fell back cannot publish plausible numbers under the wrong label.
• The edge address is resolved once per cycle and every protocol leg is pinned to it, so protocols are compared against the same anycast node.
• If the probe host is joined to the tailnet, split-horizon DNS resolves the public hostname straight to the origin and the edge route silently stops measuring the edge. The prober notices, labels the leg as tailnet-direct, and warns.

tsnet moves packets through userspace WireGuard and a userspace TCP stack. That is fine for latency and reachability, but it caps throughput: 22 to 51 Mbps on the direct route against 250 to 326 Mbps through a kernel tailscaled on the same machine. That can invert the comparison and make the edge look faster than going direct, so the prober warns at startup when the combination is active. For throughput, it runs in host mode on a machine that already has tailscaled.

Transcode probing is off by default, because each run costs the server a real transcode. When it is on, one session per cycle follows the same sequence a real client does: decision, then start, then segments. The duration and resolution limits are ceilings rather than defaults, thirty seconds and 1080p, and a larger value is rejected at startup because the cost of getting them wrong lands on the server being measured. The session is stopped on every path, success, error and timeout, and stopped before the result is recorded. The stop request deliberately does not inherit the probe’s context, since that context is routinely cancelled by the duration cap, and a stop that was never sent would leave the server transcoding for nobody. The metric that matters is the time to the first segment, which is how long before playback could have begun.

Finally, the prober can measure the cache against its own absence. With the cache’s control token it runs three reads back to back over the same window of the same file: cold, where the cache is asked to serve a window it does not hold yet; warm, the same window again; and bypassed, the same bytes with the cache told to stand aside. Results leave as Prometheus metrics pushed by remote write through the tsnet node, so Prometheus never has to find the prober.