mop_cli: The Operational Control Plane for a Multi-Tenant Rust Platform
Summary
mop_cli is a Rust command-line tool I built to serve as the single operational interface for the entire mop_dev platform — a custom Hub & Spoke infrastructure for hosting multiple web applications on a hardened Debian server. What started as a runner-setup helper evolved into a comprehensive DevOps toolkit that ships as four compiled binaries. Together they handle everything the platform needs to go from a blank server to a running, isolated, zero-downtime-deploying spoke application: GitHub account management, Docker runner provisioning, a least-privilege host-agent security broker, secrets synchronization, domain onboarding across three cloud providers, a complete i18n toolchain, CSS dead-code elimination, and more. Every piece of logic that would otherwise require manual SSH commands, web dashboards, or one-off scripts is encoded here in Rust, tested, and accessible through a consistent CLI interface.
1. Context: What the CLI Operates
To understand mop_cli, it helps to understand the infrastructure it controls. The mop_dev platform runs on a single Debian 13 server sitting behind Cloudflare. A central Hub application (mop_dev) acts as an Identity Provider and reverse proxy, routing requests by Host header to a set of spoke applications — each an independent Rust/Axum backend with a Yew WASM frontend, its own PostgreSQL database, and its own OS user identity (UID 2001–2009).
Deploying to this platform is not just git push && ssh in && restart. Each spoke runs inside a Docker container for CI builds but deploys as a native systemd service. A security broker daemon sits on the host, mediating every privileged action a container is allowed to request. Secrets are managed in GitHub Actions. Domains are provisioned across Cloudflare, AWS SES, and Google Search Console. The CSS shared across the platform needs pruning as components evolve. Locale YAML files need to stay aligned with source code and database content.
mop_cli is the tool that connects all of these moving parts.
2. Four Binaries From One Crate
The mop_cli Cargo package compiles to four binaries, each serving a distinct role in the deployment pipeline:
mop is the developer-facing CLI. It exposes all operational commands through a structured clap interface, from account management to domain onboarding to i18n tooling. This is the binary a developer invokes at their workstation.
mop-host-agent is a daemon that runs persistently on the production server. It listens on per-spoke Unix sockets under /run/mop_runner/, authenticates callers using SO_PEERCRED (the kernel-level credential of the connecting process), and executes a small set of typed host-side actions on their behalf. It is the security boundary between CI containers and the host.
mop-runner-agent runs inside each Docker CI container. It connects to the host-agent socket that compose bind-mounts at /run/host-agent.sock and forwards deployment requests — activate a release, run migrations, restart the app — without ever having direct host access.
mop-agent-cli is the thin command-line client for the host-agent protocol. It is invoked by GitHub Actions deploy.yml workflows to issue individual actions (e.g., mop-agent-cli activate-release, mop-agent-cli restart-app) and report success or failure back to the CI log.
This separation of concerns — developer tooling, host daemon, container-side agent, CI client — means no single binary has more privilege than its role requires.
3. The SPOKE_TABLE: A Single Source of Truth
A foundational design decision in mop_cli is the SPOKE_TABLE in src/spokes.rs. It is a compile-time array of SpokeIdentity structs that encodes the complete identity of every spoke on the platform:
pub struct SpokeIdentity {
pub name: &'static str,
pub uid: u32,
pub gid: u32,
pub bridge_subnet: &'static str,
pub app_unit: Option<&'static str>,
pub service_user: Option<&'static str>,
pub release_dir_prefix: &'static str,
pub allow_migrate_prod: bool,
}
Every field has operational meaning. uid and gid are the OS identities the host-agent uses to authenticate incoming socket connections. app_unit is the systemd unit the agent is allowed to restart — None for library crates like mop_shared and mop_crypto that only publish artifacts. release_dir_prefix is the path the agent canonicalizes and checks against before performing any symlink swap, defeating directory traversal attacks. allow_migrate_prod is an explicit capability flag, not an implicit permission.
This table is consumed by three distinct subsystems: the Docker compose renderer (which generates per-spoke docker-compose.yml files), the bootstrap script generator (which creates per-spoke OS users during server setup), and the host-agent ACL enforcer (which checks every incoming request against the spoke's declared permissions). Changing what a spoke is allowed to do means editing one struct field in one file — and the compiler ensures every consumer stays in sync.
The table also includes test assertions compiled into the binary: unique UIDs, unique names, unique subnets, and release prefixes that end with /. These run as standard Rust unit tests and catch misconfiguration before it reaches production.
4. The Host-Agent: Security Broker Between CI and Host
The host-agent is the most security-critical component in the codebase. The architectural problem it solves is real: a Docker CI container that builds and tests a spoke application needs to perform host-side operations at deploy time — swapping a symlink, running database migrations, restarting a systemd unit. The naive solution is to give the container sudo access or share SSH credentials. Both approaches violate least privilege and expand the blast radius of a compromised runner.
The host-agent replaces all of that with a typed, authenticated, ACL-gated protocol over a Unix socket.
Authentication happens at the socket layer using SO_PEERCRED. When a container connects to the spoke's socket, the kernel provides the connecting process's UID without any possibility of spoofing. The daemon looks up that UID in the SPOKE_TABLE and knows exactly which spoke is making the request — and what it is allowed to do.
The protocol is a tagged JSON enum with no string pass-through:
pub enum AgentRequest {
RestartApp,
ReloadSystemd,
ActivateRelease { release_dir: PathBuf },
MigrateProd { release_dir: PathBuf },
WriteSharedEnv { content: String },
}
There is no RunCommand or ExecScript variant. Every action the agent can perform is a named, typed operation. Adding a new capability requires a code change, a review, and a daemon release — it cannot happen at runtime by passing a different string.
The ACL is enforced in src/host_agent/acl.rs using a pure function that takes a SpokeIdentity and an AgentRequest and returns Ok or Err. Library spokes (mop_shared, mop_crypto) cannot call RestartApp or WriteSharedEnv because they have no app_unit or service_user. No spoke can MigrateProd unless allow_migrate_prod is true in its table entry. ActivateRelease and MigrateProd canonicalize the provided path and verify it starts with the spoke's declared release_dir_prefix — there is no way to activate a release outside the spoke's own /opt tree, even if the caller supplies a ../ path.
The audit log at /var/log/mop-host-agent.log records every request and its outcome as line-delimited JSON, providing a complete history of host-side actions performed during deployments.
5. Server Provisioning and Docker Runner Setup
The mop server command group handles the full lifecycle of runner infrastructure. The current production path is the Docker runner flow, which mop_cli fully orchestrates.
mop server bootstrap-docker runs once when setting up a host. It SSHs into the server and executes a generated bootstrap script that installs Docker, creates per-spoke OS users with UIDs derived from the SPOKE_TABLE, builds and installs the mop-host-agent daemon as a systemd service, and sets up the /opt/mop_runner/ directory tree. The script is generated in Rust from the SPOKE_TABLE — not hardcoded bash — so adding a new spoke automatically generates the correct user creation and socket provisioning commands.
mop server image-build builds the base Docker image that all runner containers share. The image contains Rust, the WASM toolchain, and everything needed to compile spoke applications.
mop server create --kind docker provisions an individual spoke runner. It generates the per-spoke docker-compose.yml and .env files from templates rendered in src/docker.rs, uploads them via SSH, and starts the container. The generated compose file hard-codes all security settings: cap_drop: ALL, no-new-privileges: true, a pids_limit of 4096, a memory ceiling, and a /tmp tmpfs with exec permission (required for Rust doctests). It also bind-mounts the spoke's host-agent socket and the /opt/mop_shared/current and /opt/mop_crypto/current directories in read-only mode.
The legacy --kind systemd path remains available for hosts that have not yet migrated, with configurable sudoers profiles (library, full-no-db, full-with-db, test-db, legacy-all) that constrain what the runner can invoke as root.
6. GitHub Account and Secrets Management
mop account add <name> provisions a complete GitHub identity: it generates an ED25519 SSH key pair stored in ~/.ssh/mop-cli/<name>/, configures a host alias in ~/.ssh/config so different accounts can coexist without key conflicts, validates the provided Personal Access Token against the GitHub API, and stores everything in a per-platform config file (%APPDATA%\mop\mop-cli\config\config.toml on Windows, ~/.config/mop-cli/config.toml on Linux). Switching between accounts with mop account switch changes which GitHub identity mop github list and mop secrets sync use.
mop secrets sync reads a .env file, filters out development-only variables (those containing localhost, 127.0.0.1, or development-specific flags), and pushes the remainder to GitHub Actions secrets via the API. It produces a .env.production artifact locally so there is always a record of what was synced. mop secrets rotate regenerates all sensitive credentials and pushes the new values. mop secrets init initializes a fresh repository with a standard set of MOP secrets, generating strong passwords at init time.
7. Domain Onboarding Automation
Bringing a new domain onto the platform used to mean logging into three separate dashboards — Cloudflare, AWS, Google — and coordinating a sequence of DNS writes, API calls, and verification handshakes. mop domain compresses this into two sequential commands.
mop domain cloudflare <domain> creates or reuses a Cloudflare zone for the new domain. Rather than applying a static template, it reads the live configuration of a reference zone (default: cuic.it, the canonical spoke configuration) over the Cloudflare API — its A records and all editable zone settings — and replicates them onto the new zone. A record names are rebased from the source domain to the target (e.g., cdn.cuic.it → cdn.example.com), the IP can be overridden, and the proxied flag and TTL are preserved. The command prints the nameservers Cloudflare assigns, which the operator sets at their registrar. --dry-run prints every intended API call without making any changes.
mop domain email <domain> handles the email and search visibility layer. It creates an Amazon SES tenant and domain identity using the aws-sdk-sesv2 crate, then writes the three DKIM CNAME records, an SPF TXT record, and a DMARC TXT record to the domain's Cloudflare zone. DKIM records are explicitly set to proxied: false because Cloudflare's proxy breaks SES verification. Optionally, it adds Google Workspace MX records. If a Google service-account JSON key is provided (via --google-sa or the config file), the command triggers a DNS-TXT ownership verification through the Google Site Verification API and registers the domain as an sc-domain: property in Search Console. All three cloud providers are orchestrated in a single command invocation, with idempotency throughout — re-running the command is safe because existing resources are detected and reused.
The DOMAIN_ONBOARDING_FUNCTION/ directory in the repo contains the research notes and prototype Rust snippets that informed these implementations: the Google Site Verification and Search Console OAuth2 flow using yup-oauth2, and the SES tenant + DKIM + Cloudflare DNS wiring.
8. The i18n Toolchain
The platform uses rust-i18n for localization, with literal source strings as locale keys. As the codebase grows, the locale YAML files can drift in both directions: new t!() calls go untranslated, and old entries linger after the strings that used them are deleted. mop_cli provides a full toolchain to manage this lifecycle.
mop tscan is the workhorse. It scans all .rs files for t!("…") macro calls using regex, compares the found keys against the existing locale YAML files, and writes one or more untranslated*.csv files containing the gaps. These CSV files can be filled in with translations and saved as translated.csv; a second run of tscan imports the file and updates the locale YAMLs without touching existing entries. The command supports several output modes: size-based chunking (splitting large translation sets into files ≤ N KB for easier handling), per-page output (one CSV per source .rs file), and a single-file scan. With --db, it also connects to both the local and production PostgreSQL databases, scanning JSONB translation fields in site_pages, products, and site_forms tables to catch database-driven content that needs translation. With --ai, missing translations are filled automatically via the Gemini API.
mop i18n-harvest takes the earlier step: finding strings in the source that should be wrapped in t!() but aren't yet. It uses the syn crate to parse Rust ASTs, extracting text nodes from html! macro blocks, user-facing HTML attributes (placeholder, title, aria-label), and thiserror error messages. It explicitly excludes SQL queries, logging macros, already-translated strings, doc comments, and internal identifiers. The output is JSON, suitable for review before wrapping the strings and feeding them to tscan.
mop locale-prune handles the reverse: removing entries from locale YAMLs that are no longer in use. It computes a keep set from two authoritative sources — t!() keys in the source code, and localized strings present in the production database — and deletes any YAML entry that appears in neither. The database scan is authoritative and mandatory by default; if it fails, the command aborts rather than risk deleting a key that is still in use but not visible in the source scan. --dry-run shows what would be removed before any files are modified. In practice, a single run of locale-prune on the mop_dev frontend removed over 18,000 orphaned entries.
mop ai-translate and mop csv-lint round out the toolchain, handling AI-assisted translation of large batches and validation of CSV files for quoting issues before import.
9. CSS Dead-Code Elimination
Hand-written component stylesheets accumulate dead rules as components are renamed or deleted. On a platform where the shared CSS under mop_dev/cdn/shared/ is served to every spoke over the CDN, a stale rule stays in every visitor's browser cache indefinitely.
mop css-deps <app> visualizes the full stylesheet dependency graph for a given frontend. It resolves Trunk-bundled local files, CDN-hosted stylesheets (which map to local paths in the workspace), and @import chains within those stylesheets, printing a tagged, ordered list of every CSS file the app loads. Tags indicate how each file was referenced ([trunk], [link], [import], [external], [generated], [MISSING]). This tool makes it straightforward to audit which files an app actually pulls in before making changes.
mop css-prune removes unused class and ID rules from every .css file across the workspace. It builds a "used" token set by scanning all source files (.rs, .html, .js, .ts) across every app root simultaneously — not per-app — so a class defined in shared CSS but referenced only by one spoke is never wrongly removed. Dynamically constructed class names (e.g., format!("fi-{}", country_code)) are handled through prefix detection: the fi- prefix keeps every .fi-* rule. Element selectors, pseudo-selectors, attribute selectors, @keyframes, @font-face, and @import are always kept. @media and @supports blocks are pruned internally and dropped only if they become entirely empty. The command is deliberately conservative: a rule is only removed when every class or ID in its selector is unused. The auto-generated CSS bundles (shared_critical.css, shared_deferred.css) are excluded from rewriting because mop_frontend/build.rs regenerates them from the component source files on each build — pruning the component sources is sufficient.
A typical run in --dry-run mode shows: DONE Would remove 521 rule(s) + 6 selector(s), 49449 bytes across 20 file(s).
10. Infrastructure and Utility Commands
Several smaller command groups complete the toolkit.
mop terraform apply generates Terraform HCL for a domain — A record, CNAME, SSL mode, HTTPS redirect, Brotli compression, auto-minify — and runs terraform init && terraform apply. This is the older provisioning path, superseded for most use cases by mop domain cloudflare but still useful for one-shot DNS setup.
mop cloudflare manages multiple Cloudflare account credentials in the config file, with the same add/list/switch/remove pattern as mop account. The active Cloudflare account is used by all mop domain and mop terraform commands.
mop server secure connects to the production server via SSH and configures UFW to deny all incoming traffic by default, rate-limit SSH, and allow HTTP/HTTPS only from Cloudflare's published IP ranges. It can be combined with mop server create --secure to harden a server in the same operation that provisions its runner.
mop artifacts provides status (total size and count of GitHub Actions artifacts) and prune (delete artifacts older than N days, with --force to execute and --dry-run to preview). This keeps storage costs bounded without manual dashboard visits.
mop structure generates a human-readable project structure tree from the workspace root, respecting standard ignore patterns (build artifacts, dependencies, version control, caches, IDEs). It writes the output to PROJECTS_STRUCTURE.md and supports --depth, --ignore, and --dry-run.
mop pwa generates PWA icons from a source image across the full set of required sizes, outputting them to the assets directory.
mop utils hash <password> generates an Argon2id hash suitable for direct insertion into the users table, printing the SQL statement alongside the hash for convenience.
11. Design Principles
Looking back at the codebase, a few consistent principles shaped every feature.
Type safety as enforcement. The host-agent protocol is not a string interface — it is a typed Rust enum. The SPOKE_TABLE is a compile-time array, not a configuration file read at runtime. The SpokeIdentity struct carries explicit capability flags (allow_migrate_prod, presence or absence of app_unit) that feed directly into ACL checks. The compiler enforces consistency between definitions and consumers.
Idempotency by default. Every domain onboarding command detects existing resources and reuses them rather than failing or creating duplicates. Every --dry-run flag is a first-class path, not an afterthought. Running a command twice should always be safe.
Least privilege, structurally. The four-binary architecture ensures no single binary combines developer convenience with host-level privilege. The Docker compose template hard-codes all security settings so they cannot be weakened by per-spoke customization. The ACL check function is a pure function with no I/O, making it easy to unit test and reason about.
Conservative over aggressive. The CSS pruner keeps rules when in doubt. The locale pruner aborts if the database is unreachable. The secrets syncer filters out development variables. Every tool that rewrites files supports --dry-run and generates a diff for review before applying.
Conclusion
mop_cli is, at its core, the artifact of a discipline I developed while running a multi-tenant platform solo: the discipline of encoding operational knowledge into code instead of documentation. Every command in the CLI represents a procedure that once required manual steps — SSH sessions, web dashboards, API calls coordinated by hand. Moving that knowledge into Rust means it is tested, reproducible, and available to any future contributor or collaborator without requiring them to reconstruct the same operational understanding.
The host-agent, in particular, represents the clearest expression of that philosophy: not just automating a task, but designing the automation so that doing it the wrong way is structurally impossible.