Rebuilding mop_dev: From a Clever Monolith to a Partitioned, Self-Rendering Platform

A while ago I wrote about building mop_dev, my custom Rust Hub & Spoke infrastructure for hosting high-end, unconventional web apps. That version worked — but writing software in production is the fastest way to discover where your first design was naive. Over the last few months I tore into the foundations and rebuilt the parts that were holding the platform back.

This is the follow-up. It's an honest account of what changed, what I got wrong the first time, and the engineering behind the new version: a least-privilege deployment broker, true per-tenant OS isolation, full server-side rendering with hydration, and a move away from one of the cleverest ideas in v1.

The bones are the same — a hardened Debian 13 box behind Cloudflare, a central Hub that acts as Identity Provider and reverse proxy, and a constellation of spoke applications routed by Host header through an O(1) HostSwitchService. What changed is almost everything underneath.

1. Deployment: from raw runners to a host-agent broker

In v1, every spoke was deployed by a self-hosted GitHub Actions runner that had a frightening amount of power: it could sudo systemctl, write .env files, swap symlinks, run migrations — all as a shared service user. It worked, and the atomic symlink swap gave me zero-downtime deploys. But a compromised runner, or a single bad workflow, had the keys to the whole machine.

The new version puts a wall between CI and the host. Each spoke now builds inside its own Docker runner container (compose stack under /opt/mop_runner/<spoke>/), and that container cannot touch the host directly. Instead, every privileged action — activating a release, running production migrations, restarting the service — is sent as a typed request to mop-host-agent, a small broker running on the host.

The agent listens on a per-spoke Unix socket (/run/mop_runner/<spoke>.sock), authenticates the caller with SO_PEERCRED, and checks the request against a hard-coded ACL. The single source of truth for that ACL is a SPOKE_TABLE in Rust:

SpokeIdentity {
    name: "womed_it",
    uid: 2002,
    gid: 2002,
    bridge_subnet: "172.30.2.0/24",
    app_unit: Some("womed.service"),
    service_user: Some("womed_it"),
    release_dir_prefix: "/opt/womed/releases/",
    allow_migrate_prod: true,
},

So the womed runner can only ever activate a release under /opt/womed/releases/ (the agent re-checks the path after canonicalization, defeating ../ tricks), can only restart womed.service, and can only migrate its own database. A deploy now looks like this:

build in container
  → stage into /opt/<spoke>/releases/<version>/
  → mop-agent-cli migrate-prod      # agent runs it, reading the operator-managed .env
  → mop-agent-cli activate-release   # agent does the atomic symlink swap
  → mop-agent-cli restart-app
  → health check → auto-rollback to previous release on failure

The zero-downtime symlink swap I was proud of in v1 is still there — it just moved behind the agent, where the runner can request it but never perform it. Production secrets followed the same principle: the .env at /opt/<spoke>/shared/.env is now operator-managed and never written by CI. The container doesn't even read it.

The migration is nearly complete: mop_dev, mop_shared, mop_crypto, womed_it, cuic_it, mop_template, mop_timistic, and factorio_index all run the broker flow. One spoke (mop_games) is still on the legacy path — a reminder that real migrations are rolling, not flag-day events.

2. True per-tenant isolation

The most important security change isn't visible in any feature: it's identity. In v1, every spoke service ran as the same shared mop user. If one tenant app was popped, the blast radius was every other tenant's files and database access.

In the new version, every spoke is its own OS principal. Each has a dedicated system user and UID (2001–2009), owns its own /opt/<spoke> tree, and — for the containerized runners — sits on its own isolated Docker bridge subnet (172.30.x.0/24). The systemd unit for each service hard-codes its User=/Group=, and that pairing is exactly what gives each tenant a distinct runtime identity.

The service hardening from v1 carries over and gets sharper:

NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/mop/shared/logs /opt/mop/shared/keys
ReadOnlyPaths=/opt/mop/current /etc/ssl/mop
AmbientCapabilities=CAP_NET_BIND_SERVICE   # only the Hub needs to bind 443

Only the Hub gets CAP_NET_BIND_SERVICE; the spokes bind unprivileged high ports on loopback and are never reachable from the public internet. The TLS material in /etc/ssl/mop is owned by the Hub user alone — the other spokes terminate TLS at Cloudflare and never see a private key. It's defense in depth that finally matches the multi-tenant claim the architecture was always making.

3. The big one: full SSR with hydration

v1 shipped a Yew WASM frontend that rendered entirely on the client. Fast once loaded, but it paid the classic SPA taxes: a blank shell until the wasm booted, mediocre Largest Contentful Paint, and SEO that depended on crawlers executing JavaScript.

The new Hub does full server-side rendering with client hydration. The Axum backend links directly against the frontend crate (crate-type = ["cdylib", "rlib"]) and calls its server renderer for every route, injecting real markup into the shell. The client then calls Renderer::hydrate() to attach to that markup instead of repainting it. Per-route <title> and canonical tags are emitted server-side, and the request's auth cookie and Host are threaded into the render so authed and per-subdomain pages come back correct on the first byte.

This is the part of the rebuild I'm proudest of, because SSR in a hydrating WASM framework is genuinely treacherous and I learned it the hard way. A few of the rails I had to build:

SSR-safety. Off-wasm, calls like web_sys::window() read globalThis, which doesn't exist on the server — they don't error, they abort the process. Anything browser-derived has to be guarded with #[cfg(target_arch = "wasm32")] and applied in a post-mount effect. This matters doubly for shared components: one unguarded widget breaks SSR for every app that renders it.
Structural hydration matching. Yew matches the server DOM to the client VDOM by component-nesting shape. A structural mismatch doesn't warn — it panics during hydration and kills the whole module, leaving a page that looks fine but is completely dead. The fix is discipline: the server and client component trees must be structurally identical, and any browser value that changes structure has to be passed into the server render explicitly.
It never returns a 5xx. An earlier, unguarded SSR rollout once took the site down with a wave of Cloudflare 520s. The new engine treats SSR as an enhancement, not a dependency: a render that panics or blows past a 3-second budget falls back automatically to the static CSR shell, and an SSR_ENABLED env flag is a hard kill switch. That outage class cannot recur.

The whole playbook is documented so spokes can adopt SSR one at a time, with shared fixes landing once in the common component crate.

4. Killing my darling: goodbye PGLite, hello real Postgres

In the v1 article I enthused about a PGLite bridge — PostgreSQL compiled to WASM, with a Node.js TCP shim spinning up a memory-backed database per app for local testing. It was a cool trick. It was also a source of subtle divergence between local and production behavior, and a second database engine to reason about.

The new version makes a deliberately boring choice: real PostgreSQL everywhere. Locally you run an actual Postgres 16 with per-app users and databases — identical to production. For environments without a local Postgres (CI, quick checkouts), DATABASE_URL points at a set of real remote test databases running on the production box, isolated from production data, each user scoped to its own database. The provisioning lives in its own repo and deploys like everything else.

The lesson: "clever, and almost like production" lost to "slightly heavier, and exactly like production." Tests still run single-threaded against a shared test DB, but they now exercise the same engine, the same SQL, and the same sqlx compile-time query checks that run in prod.

5. Shared crates that deploy on their own

As the workspace grew to 31 members, copy-pasting common code stopped scaling. The new version extracts the reusable surface into mop_shared — universal WASM-safe types, backend utilities, an OAuth client library, and a set of shared Yew components (including the sign-in button every spoke uses).

What makes it interesting operationally is that mop_shared deploys independently, on the same release-and-symlink pattern as a service. Pushing to it stages a new release and atomically swaps /opt/mop_shared/current; each spoke symlinks that path at build time. No service restart for the shared deploy itself — but every spoke that depends on a changed crate must be rebuilt to pick it up, because the symlink resolves at compile time, not runtime. It's the kind of sharp edge worth documenting loudly.

6. Post-quantum crypto, baked in

The new platform integrates mop_crypto, a standalone post-quantum library (ML-KEM-1024 for key encapsulation, ML-DSA-87 for signatures, XChaCha20-Poly1305 for symmetric encryption, SHA-3/SHAKE for hashing). The Hub provisions its keypairs on first boot into a persistent key directory and reuses them across deploys, so sessions and at-rest encrypted data survive releases. Building for the threat model of a few years out, today, rather than retrofitting it under pressure later.

7. OAuth, hardened against itself

"Sign in with mop.dev" is the connective tissue of the whole ecosystem, and in v1 it had a recurring failure mode: spoke OAuth client rows were sometimes seeded by database migrations, which let the Hub's view of a client drift from what the spoke actually had deployed — silently breaking sign-in.

The new rule is strict and enforced by CI: OAuth client credentials live with the spoke (its env and GitHub Secrets), never in a Hub migration. A guard in the Hub's deploy workflow rejects any new migration that writes to oauth_clients. On the frontend, every sign-in goes through a single shared component that redirects to the spoke backend's /login route — the only place that sets the CSRF state cookie — instead of linking straight to the IdP, which was the historical cause of "state validation failed" errors. The shared mop_auth library handles the backend-to-backend token exchange, transparently routing internal calls and injecting the right Host header for the multi-tenant router.

8. Modernized foundations

Underneath the features, the whole dependency base moved forward — axum 0.8.1, tokio 1.42, sqlx 0.9, yew 0.23 — standardized once in a root Cargo.toml and shared across all 31 workspace members. Frontend performance got its own standards pass: consolidated CSS to kill render-blocking round-trips, deferred non-critical styles, and responsive <picture> hero images. Small things, compounding.

What the rebuild taught me

Three themes run through every change above. Least privilege — nothing should be able to do more than its job, from the deploy runner to the tenant service to the OAuth client. Local equals production — every place where I'd accepted "close enough" eventually cost me, so I closed the gaps. And enhancements must fail safe — SSR, in particular, taught me that a feature which can take down the site isn't a feature, it's a liability until you've built the rails that make its worst case a graceful degradation.

mop_dev v1 proved the architecture. v2 made it something I'd trust with other people's data. The art is still writing high-performance Rust on top of it — but now the foundation is honest about what it's protecting, and from whom.