Canvas

Services

The decision worth arguing about

Two decisions define this deployment, and both come from the same arithmetic. A board with a dozen active participants generates cursor and awareness updates at roughly thirty a second per person; each one must reach the other eleven. That is a few hundred inbound messages a second becoming a few thousand outbound, sustained for the length of a workshop — which is why API Gateway WebSockets, billed per message and per connection-minute, is not a candidate at any price, and why the relay is a Fargate service holding sockets directly. The genuinely non-obvious decision is the second one: how clients of the same board find the same relay node. The default answer is to let the load balancer distribute connections however it likes and have nodes exchange updates through Redis pub/sub. That works, and it quietly multiplies your internal message volume by the number of relay nodes — every one of those few thousand messages a second gets published once and delivered to all eight relays, whether or not they hold a participant on that board, so the Redis node becomes the bottleneck long before the relays do and you end up paying for a large cache instance to shuttle messages between processes that did not need to talk. Canvas instead keeps a board-to-node assignment in Redis, and a small router service tells a joining client which relay hostname to connect to, so every participant on a board lands on one process and cross-node traffic is zero. The trade-off is a real one and it shows up during deployments: if a relay node dies or is replaced, every board it held has to be reassigned and every client reconnects, which is a two-to-three-second interruption where the pub/sub design would have degraded more gracefully. We mitigate it with long deregistration delays and by draining boards to a new node between updates rather than at connection close, and we accept the blip because a whiteboard that hiccups for two seconds during a deploy is better than one that needs a $400-a-month cache to talk to itself. The third thing worth naming is CRDT growth: a Yjs document accumulates its full edit history, so a board worked on for a year is several megabytes of update log and slow to load. A compaction worker periodically writes a merged snapshot to S3 and prunes the log behind it, which costs us fine-grained time travel beyond the snapshot horizon — you can restore to any of the retained daily snapshots, not to any individual keystroke from eight months ago.

Services

The decision worth arguing about

Azure offers a managed answer to the fan-out problem — Azure Web PubSub — and the reason we do not use it is worth spelling out, because it looks like the obvious choice. Web PubSub is billed in units, and each unit carries both a concurrent-connection allowance and a daily message allowance. A whiteboard's load is wildly asymmetric between those two: a hundred and twenty concurrent editors is a trivial connection count, but the cursor and awareness traffic they generate burns through message quota at a rate that forces you to buy units purely for messages, most of whose connection capacity you will never touch. You end up scaling a managed service on the axis it prices worst. Running the relay ourselves on Container Apps costs compute we can size to the actual socket count. That choice creates the Azure-specific problem, and it is a real one: Container Apps' default HTTP scale rule counts concurrent requests, and a WebSocket held open for a three-hour workshop is exactly one concurrent request. Left on defaults, a relay node holding two thousand sockets looks idle to the autoscaler while a node holding twenty looks the same, and scale-in will happily terminate a revision full of live boards. So the relay runs on a workload-profiles environment with a custom KEDA scaler reading a Redis-held active-connection gauge that the relay publishes itself, and with a long termination grace period plus an explicit drain endpoint that migrates board assignments before the process exits. The cost of that is the workload-profiles baseline, which is why the Azure figure below sits above the AWS one; the benefit is that scale-in stops being an outage. Snapshot and update-segment storage behaves as it does on AWS, with the one caveat covered in Watchtower's notes: Blob's cool-tier early-deletion penalties make aggressive lifecycle rules a false economy on data the compactor rewrites, so snapshots stay in Hot until they are outside the compaction window.