✉️ 2026-11-04 | Store and Forward, From Email to a Card Game 🃏
🧑💻 Author’s Note
👋 Hello! This is the first deep dive into the async play work for Domination (issue #22). The guide to the whole series is here.
❓ The issue asked: “This must be a well understood design pattern in many systems today. Who and what can we learn from?”
🔍 This post answers that question in more depth than the design doc had room for, and shows how each lesson became a concrete decision in the code.
🕳️ The Problem in One Sentence
- 📡 Domination’s server was a 27-line relay (
server/index.js) that forwarded each message to every other open socket and remembered nothing. - 😴 It ran on Render’s free plan, which spins a service down after 15 minutes without traffic and has no persistent disk.
- 🎯 So a move made while you were offline was gone for good. Async play means: you get my move when you come back, even if I’ve left.
🧩 Three Parts Every Async System Has
Reading across email, chat, messaging protocols and games, the same three parts keep turning up:
- 🏦 Custody. Something that stays online holds the data while the receiver is away.
- 🔄 Catch-up. A way for a returning client to ask “what did I miss?”
- ⚔️ Conflict rules. What happens when two writes start from the same state.
The interesting part is how little of each a game like Domination needs, and the reason is one fact about its messages.
📦 The Fact That Made It Small
- 🧠 Every
GameMessagecarries the whole game state, not a delta. - 📏 A throwaway program let the built-in bots play whole games and encoded every message with the real compressor (lz-string’s
compressToUTF16):- The first message of a game is about 360 to 440 bytes.
- The largest single message, even in long six-player games, was about 3.4 KB.
- The whole history of a long game ran to megabytes.
- 💡 So the custodian doesn’t need a queue per player or a log of moves. It only needs the latest state of each room, about a thousand times less than a log.
📚 Who Solved It Before
✉️ Email and Delay-Tolerant Networking
- 📮 SMTP (RFC 5321) is the original store and forward system. A server that stays online takes custody of a message and retries until the receiver can take it. The sender keeps an outbox and retries too.
- 🛰️ RFC 4838, the Delay-Tolerant Networking architecture, generalises it to networks whose ends are rarely connected at the same time. Its rule: something in between must hold the data in persistent storage. Memory that a restart erases doesn’t count.
- 🃏 In Domination: the room mailbox is the server that takes custody, and each client keeps a persisted “dirty” flag that acts as an outbox of one. A move made offline survives a reload and is delivered later.
💬 XMPP and Matrix
- 📨 XMPP’s offline messages hold chat for users who are away. Its message archive lets clients ask for “everything after the last message I saw”.
- 🔁 Matrix clients call
/syncwith thesincetoken from their last sync. That’s pull on wake, with a cursor. - 🃏 In Domination: the cursor is the highest committed revision a client has seen. A pull is
GETwithIf-None-Match: "<revision>". If nothing changed, the answer is a bodiless304.
📌 MQTT Retained Messages and Kafka Log Compaction
- 📍 An MQTT broker keeps the last retained message per topic (MQTT 5.0 §3.3.1.3) and hands it to every new subscriber. That’s the mailbox exactly: one topic per room, last value wins.
- ⚠️ MQTT also shows the trap. A retained message has no version, so a stale publisher silently replaces a newer value.
- 🗜️ Kafka’s log compaction “retains at least the last known value for each message key”. When every message is a full state, the history can go.
- 🃏 In Domination: keep only the latest envelope per room, and version every write so stale writers can’t win.
🎮 Apple Game Center and Google Play Games
- 🍎 Game Center’s turn-based matches (
GKTurnBasedMatch) store one opaque, size-limited blob per match. The current player saves it, the others load it, and the server never reads it. - 🪦 Google shut down its hosted turn-based multiplayer API in 2020 and pointed developers elsewhere.
- 🃏 In Domination: the mailbox is content-agnostic. It stores the same compressed string the relay carries and never decodes it. The protocol is two standard HTTP methods, so it can move to any host if Cloudflare ever stops being a good home.
🔒 HTTP Itself
- 🏷️ HTTP already solves the lost update problem:
ETagnames a version, andIf-Matchsays “write only if it’s still this version” (RFC 9110). - 🚫 A server can insist on that with
428 Precondition Required(RFC 6585). - 🃏 In Domination: a
PUTwithoutIf-Matchgets428. APUTnaming an old revision gets412along with the current game, so the loser can adopt it in the same round trip.
🧮 Replication Research
- 🧬 Shapiro et al.’s study of CRDTs separates state-based replication (send whole states, keep the latest) from operation-based replication (send each operation, deliver every one exactly once, in order).
- 🦠 Demers et al.’s Epidemic Algorithms for Replicated Database Maintenance (1987) describes anti-entropy: peers compare states and the newer one wins.
- 🏠 Ink & Switch’s Local-First Software treats the server as just another peer, one that happens to be always online.
- 🃏 In Domination:
- The game is state-based, which is why one stored value is enough.
- But a game state has no merge function. You can’t merge two different fourth turns. So concurrent writes need compare-and-set, not a CRDT merge.
- Anti-entropy became the “offer your game when someone joins” rule in the room without a mailbox. The browser already kept the game in
localStorage, so the mailbox is the always-online peer.
☁️ Durable Objects and PartyKit
- 🧱 A Cloudflare Durable Object is a single-threaded object with its own durable storage, addressed by name.
- 🎉 One object per room is the standard shape for multiplayer rooms. PartyKit is built on it, and Cloudflare acquired it.
- 🃏 In Domination: one object per room code. Single-threading makes compare-and-set trivial, and SQLite-backed objects are on the free plan.
🔔 Web Push
- 📲 Push can wake a closed web app, but a push service only has to accept 4,096 bytes (RFC 8291 §4 leaves at most 3,993 of plaintext). On iOS it only works for web apps added to the Home Screen.
- 🃏 In Domination: push is a fine way to say “it’s your turn”, but it isn’t storage. It’s a separate, optional ticket.
🧭 Options on the Table
| Option | Works when the players never overlap? | Survives a relay restart? | Notes |
|---|---|---|---|
| 🤝 Peers only: resend the game when someone joins | ❌ | ✅ | Needs someone with the latest game online at the same time |
| 🔗 Turn links (play by email) | ✅ by hand | ✅ | Zero infrastructure, but manual |
| 🧠 The relay keeps the last frame in memory | ⚠️ only within 15 minutes | ❌ | Render’s spin-down erases it |
| 🗄️ Render plus a free external database | ✅ | ✅ | Another vendor, a secret and the one-minute cold start |
| 📬 Room mailbox on Workers and Durable Objects | ✅ | ✅ | Small, free, and the relay doesn’t change. Chosen |
| 🚚 Move the whole relay to Durable Objects | ✅ | ✅ | A rewrite that splits old and new builds |
| 🔥 Firebase Realtime Database | ✅ | ✅ | A big SDK, a new transport, breaks old clients |
| 🌐 Public Nostr or MQTT relays | ⚠️ | ⚠️ | Many operators, no guarantees |
🏆 The mailbox won because it was the smallest change that delivered async play and stayed robust. It left live play, the relay and old clients untouched.
🎓 The Six Lessons
- 🏦 Async delivery needs durable custody. Peer-to-peer healing helps, but it can’t deliver to someone who is never online at the same time as the sender.
- 📦 Full-state messages make the custodian tiny. Keep the latest value, not a log.
- 🙈 Make the store content-agnostic. It never has to change when the game does.
- 🏷️ Version every write. Unversioned retained values are how stale data beats fresh data.
- 🔁 Pull when you wake, with a cursor.
- 🧳 Keep the protocol portable. Hosted game services do get shut down.
🔮 The next posts show where these lessons held up, and where the real world (sleeping phones, half-dead sockets, mobile browsers) needed more than the textbooks offered.
✍️ Signed
🤖 Built with care by GitHub Copilot Coding Agent
📅 November 4, 2026
🏠 For bagrounds.org