Performance Engineering on mop_dev: From 20-Minute Deploys to a Sub-Second Desktop Experience

Tags: Rust · WebAssembly · Axum · Yew · Performance Engineering · Web Vitals
Period: June 2026

Summary

Over a focused two-week sprint I systematically profiled and optimized every performance-critical layer of mop_dev — a full-stack Rust web platform built on Axum (backend) and Yew/WASM (frontend). The work spanned backend latency, WASM binary size, frontend loading, and mobile Web Vitals.

The headline results:

Metric	Before	After
CI deploy time	~20 min	~6 min
Shipped WASM (raw)	27.6 MB (debug) → 6.43 MB	4.96 MB
Shipped WASM (brotli)	~1.5 MB	1.01 MB
Render-blocking CSS budget	1,380 ms	150 ms
CLS (homepage)	0.055	0.004
Backend: per-request DB queries	1 query + new TCP per request	Cached, reused pool
Request log throughput	1 INSERT/request	Batched, 100-row bulk inserts

No feature was removed, no user-visible behavior changed, and every change shipped behind a clean cargo clippy -D warnings + test suite pass.

1. Backend: Eliminating Per-Request Waste

1.1 Dynamic Proxy Cache — Biggest Single Backend Win

The platform routes traffic for multiple tenants (e.g. womed.it, cuic.it, factorioindex.com) through a dynamic proxy layer. Before the fix, every single request on a tenant domain triggered:

SELECT upstream_url FROM proxy_routes WHERE host = $1 — a database round-trip
build_proxy_router(&upstream_url) — which constructed a brand new reqwest::Client (and therefore a brand new connection pool), immediately discarding any keep-alive connections from the previous request

The fix: a host → Arc<ProxyState> cache in an Arc<RwLock<HashMap>> with a 30-second TTL and a 1,024-entry bound. Negative results (unknown hosts) are also cached, so invalid Host header spraying can't drive unbounded DB load. The TTL means admin edits to proxy_routes propagate within 30 seconds — no manual cache flush needed. A hand-rolled cache kept the implementation dependency-free.

Effect: tenant request latency drops by one DB query + TCP handshake per request. Under load (80 req/s at the rate-limiter cap) this was previously 80 database roundtrips/second just for proxy routing.

1.2 Database Pool Settings

A one-line configuration bug had idle_timeout(1s) and min_connections(0). Every connection that sat idle for more than a second was destroyed, meaning the next request would pay PostgreSQL's full TCP + auth setup (~5–10 ms). Changed to min_connections(2) and idle_timeout(300s), both tunable via environment variables (DATABASE_MIN_CONNECTIONS, DATABASE_IDLE_TIMEOUT_SECS).

1.3 Batched Request Logging

The request-logging middleware was spawning a tokio::task and firing an individual INSERT INTO request_logs for every request — up to 80 inserts/second competing directly with application queries. The codebase already had the right pattern in the analytics subsystem: a bounded mpsc channel drained by a batch worker.

The rewrite routes log entries through a bounded 4,096-slot channel. A background worker (run_request_log_worker) drains it and bulk-inserts via QueryBuilder every 100 rows or 1 second, whichever comes first. The channel is best-effort: a full queue drops the entry rather than blocking the request path.

Two follow-up fixes landed alongside this: a static-asset sampling switch (successful 2xx/3xx responses for WASM/JS/CSS/fonts are sampled 1-in-10, configurable) to cut the dominant low-value log volume, and a fix for a pre-existing double-log bug on the non-TLS path where the middleware was layered twice.

1.4 Precompressed Static Asset Serving

The Axum ServeDir serving the Trunk build output (*_bg.wasm, JS, hashed CSS at the dist root) had no precompression configured. The /assets path had it; the root fallback didn't.

Added .precompressed_gzip().precompressed_br() to the root fallback ServeDir, the /cdn nested serve, and the cdn.mop.dev ServeDir. A new deploy step runs brotli -q 11 and gzip -9 on all compressible static types in dist/ and cdn/ before bundling, so precompressed siblings ship with every release. ServeDir only serves a sibling when the client's Accept-Encoding matches and falls back to the raw file otherwise — safe to add with no client-side changes.

Result: the WASM (then ~6 MB) was compressed to ~1.5–2 MB brotli at the origin, guaranteeing correct encoding even if Cloudflare's dynamic brotli for application/wasm is inconsistent.

1.5 Proxy Header Cloning

infra/proxy.rs was cloning the full request header map and the full upstream response header map, once each, per proxied request. The fix uses req.into_parts() to move method and headers out (zero copy on the request side), and on the response side iterates the upstream headers by reference into a fresh HeaderMap rather than cloning the entire map first. Minor on its own, but illustrative of the layered approach: no allocation is free.

2. WASM Binary Size: 27.6 MB → 4.96 MB

The most visible performance number for a Yew/WASM frontend is the cold-load binary size. mop_dev's shipped WASM went through four major reduction passes.

2.1 Locale Embedding — Largest Single Lever (−multi-MB, deploy time −14 min)

The rust-i18n macro was embedding all 7 supported locales at compile time. Each locale is a YAML file; together they accounted for ~6.5 MB of embedded strings and ~22,000 generated initializer statements per codegen unit. This was also the primary driver of the 20-minute CI deploy time (trunk's codegen for the i18n! macro is proportional to the number of embedded keys).

Solution: a mop_frontend/build.rs splits the locales/ directory into an embed set (only en + it) and a runtime set (fr/de/es/zh/ar). The runtime locales are compiled to flat JSON by build.rs during the build, served from cdn.mop.dev, and fetched lazily on first language switch by a custom rust-i18n backend (i18n_runtime.rs). The macro now generates ~6,400 statements instead of ~22,000.

A reliability fix came alongside this: because build.rs writes locale JSON to ../cdn as a side effect, Cargo won't re-run it if locales/ is unchanged in a fresh checkout. Fixed by injecting MOP_DEPLOY_NONCE = commit SHA to force a rebuild every deploy, plus a deploy guard that fails the job if any runtime JSON is missing.

Effect: CI deploy time ~20 min → ~6 min; WASM codegen unit shrank dramatically.

2.2 i18n Init Closure → Static Lookup Tables (−336 KB)

Even after the locale count was reduced, the i18n! macro's add_translations closure pattern — one closure call per key per locale at startup — contributed ~646 KB of WASM code. The replacement: build.rs emits sorted &[(&str, &str)] slices to OUT_DIR/i18n_static.rs; the custom I18nBackend binary-searches them at runtime. No runtime HashMap allocation at startup, no macro-generated closure tree in the binary.

Result: −336 KB raw WASM; the closure is also gone from the native SSR binary, saving startup time server-side as well.

2.3 Article Components Extracted to CDN Markdown (−1.3 MB code, deploy: 6.43 → 6.01 MB)

The 14 article pages were implemented as Yew html! components — Rust source files that compile to WASM code. Together they accounted for ~1.3 MB of compiled WASM code (stake-roobet 252 KB, ai-containment 226 KB, pirate-streaming 213 KB, etc.), entirely for what is essentially static authored content.

The refactor:

Article content moved to per-language markdown under cdn/mop_main/articles/<slug>/<lang>.md
Backend features::articles renders markdown via pulldown-cmark, sanitizes with ammonia (same allowlist as the legal service), and serves via GET /api/v1/articles/{slug}?lang=
On SSR, the article body is pre-rendered into the HTML shell via a new SsrArticleHtml context (same injection mechanism as #site-config), so first-paint is instant with no client roundtrip
On SPA navigation or language switch, the frontend fetches the pre-rendered HTML via the API — no Markdown parser in the WASM
The 13 html! component files and the slug→component dispatcher were deleted

A new mop tscan --articles mode automates translating source Markdown into the 6 missing languages (chunked, Gemini model-rotation, with structural QA checking heading counts, code fences, and content-length ratios to catch AI truncation).

A companion mop tscan --prune mode removes locale keys no longer referenced by any t!() call in source. Running it after the article extraction reclaimed ~0.9 MB of orphaned article text from the .rodata section (locale YAML bodies that were still embedded even after the component files were deleted): en.yml shrank from 4,195 keys to 1,534.

2.4 `ammonia` Removed from WASM (−0.5–1 MB)

The HTML sanitization library ammonia (which pulls in html5ever and markup5ever) was being included in the WASM build for a single call site in the legal page. Moving sanitization to the backend legal service allowed the frontend to call Html::from_html_unchecked on pre-sanitized HTML from a trusted endpoint, removing the entire ammonia/html5ever/markup5ever dep tree from the WASM graph.

2.5 `reqwest` Removed from the WASM Graph (−220 KB raw / −74 KB brotli)

An initial twiggy top profile showed only ~61 KB of direct reqwest symbols in the WASM, which led to an early decision to skip this lever. Revisiting with a focus on the dep tree rather than direct symbols revealed the real cost: removing reqwest from the [dependencies] (not just dead-code-eliminating it) also evicts its transitive chain — url, idna, http, percent-encoding — which LTO can then eliminate entirely.

Six dual-target files (logic shared between SSR prefetch on the native side and client-side fallback on WASM) were using reqwest directly. The fix: a new mop_frontend/src/utils/net.rs with cfg-split helpers (get_json / get_json_bearer / post_json) that use gloo-net on wasm32 and reqwest on native. reqwest moved to [target.'cfg(not(target_arch="wasm32"))'.dependencies] to make it structurally impossible for it to re-enter the WASM build.

Measured result: −220 KB raw / −74 KB brotli — 3.5× the direct-symbol estimate.

2.6 WASM-Only Release Profile (−587 KB raw / −60 KB brotli)

The standard Rust advice for size-optimized WASM — lto="fat", codegen-units=1, opt-level="z", panic="abort", strip=true — was previously ruled out because mop_dev's [profile.release] is shared with the Axum backend, and panic="abort" there turns any request-handler panic into a whole-process abort that tower's catch_panic middleware cannot intercept.

The solution: a separate [profile.wasm-release] in mop_dev/Cargo.toml that inherits from release and adds the aggressive flags. Applied only in the deploy command:

trunk build --release --cargo-profile wasm-release

Critically, it is not set in Trunk.toml [build] cargo_profile — that key applies to trunk serve as well, which would force fat-LTO rebuilds on every hot-reload during development. Backend builds use cargo build -p mop_backend --release and never touch this profile.

Effect: −587 KB raw / −60 KB brotli. Full wasm-release build is ~2m11s locally — acceptable since the deploy builds the frontend exactly once.

Cumulative WASM Size

Stage	Raw	Brotli
Original (debug artifact)	27.6 MB	—
CI release baseline	~6.75 MB	~1.5–2 MB
+ locale split (§2.1)	6.43 MB	—
+ i18n static tables (§2.2)	6.43 MB → 6.01 MB	~1.19 MB
+ article extraction (§2.3)	6.01 MB	—
+ wasm-release profile (§2.6)	5.43 MB	1.13 MB
+ reqwest removed (§2.5)	4.96 MB	1.01 MB

The original 3–5 MB raw target is met. Further gains require pruning orphaned locale .rodata (~0.9 MB via mop tscan --prune) or route-level code-splitting.

3. Frontend Loading: Render-Blocking and Lazy Loading

3.1 Render-Blocking Third-Party Scripts Removed

index.html was loading <script src="https://unpkg.com/html5-qrcode"> (367 KB) and Leaflet (145 KB) synchronously on every page, blocking render even on pages that use neither. Both libraries were also sourced from unpkg.com, which carries no SLA.

Fix: both are now lazy-injected by their components on mount — qr_scanner.rs calls ensure_html5qrcode_loaded(), leaflet_map.rs calls ensure_leaflet_loaded() — and removed from index.html. They are vendored at pinned versions under cdn/mop_main/vendor/ and served from cdn.mop.dev with immutable cache headers. unpkg.com was removed from the CSP entirely.

Effect: 512 KB of JavaScript removed from the critical path of every page that doesn't use those features.

3.2 CSS `@import` Waterfall Eliminated (−1,380 ms render-blocking budget)

cdn/shared/shared_components.css was a hub file that @import-ed 22 sub-files. CSS @import is both render-blocking and serialized: the browser must download the hub, parse it, discover the imports, and then fetch each one in sequence — a chained waterfall that defeats HTTP/2 multiplexing. Lighthouse measured ~34 render-blocking requests and an estimated 1,380 ms of savings.

Additionally, the hub file dragged admin/* and features/* CSS onto the critical path of every anonymous homepage visitor.

The fix generates two concatenated, @import-free bundles via mop_frontend/build.rs (generate_css_bundles):

shared_critical.css — utilities + all ui/* chrome. Render-blocking (needed before first paint).
shared_deferred.css — features/* + admin/*. Loaded non-blocking via media="print" onload="this.media='all'" with a <noscript> fallback.

index.html references the two bundles; the original shared_components.css is left untouched for other spokes that depend on it. The bundles are generated by build.rs on every build (gitignored), so trunk serve, trunk build, and the deploy all stay in sync cross-platform — no shell scripts required (which would break on Windows dev machines or Linux CI runners respectively).

This also fixed the CLS div.hero-cta regression: the hero CTA buttons were rendered unstyled until the late-arriving buttons.css import resolved, causing a layout shift (CLS 0.055). With buttons in the render-blocking critical bundle, they are styled at first paint.

3.3 Cal.com Embed Removed

index.html ran Cal("init") on every page, loading app.cal.com/embed/embed.js — but the site only uses plain <a> links to cal.com, not an inline booking widget. The embed was dead weight that:

Loaded Cloudflare's bot-management main.js (source of Lighthouse's "deprecated Privacy Sandbox APIs" warning)
Set a __cf_bm third-party cookie on every page visit
Ran on every page including admin and tenant spokes

Removed the embed <script> and tightened the CSP (cal.com/app.cal.com dropped from script-src, connect-src, frame-src). The  SSR markers and SW registration were left intact; the explanatory comment avoids naming the vendor domain to stay white-label-safe.

3.4 Anonymous `/auth/me` 401 Noise

The navbar probed /auth/me on mount for cross-subdomain cookie hydration. For anonymous visitors (no session cookie) this produced a browser-logged 401 on every page load — a Lighthouse Best Practices finding and unnecessary server load.

Fix: a non-sensitive, JS-readable mop_authed=1 hint cookie (Domain=.mop.dev) is set alongside the HttpOnly auth_token on login and cleared on logout (with self-heal on a 401 response). The navbar now checks is_authenticated() || has_session_hint() before probing. The HttpOnly auth_token remains the sole credential — the hint only suppresses the pointless anonymous probe.

3.5 Non-Critical CSS Bundles Deferred

After the @import waterfall was fixed, Lighthouse flagged a remaining ~350 ms of render-blocking from Trunk's own hashed CSS bundles. Audited each:

animations.css → deferred. The hero's animate-fade-in class is a no-op (undefined), and the reveal classes it defines are unused above the fold. Deferred via media="print" onload.
presentation.css → deferred on apex, restored blocking on tenants. Used only by multi-tenant block-site pages — dead weight on the marketing homepage. Deferred in the shell; the SSR tenant path (apply_presentation_head) strips the defer back to a blocking <link> for tenant hosts to avoid FOUC.

A subtle footgun was caught here: Trunk HTML-entity-encodes onload attribute values in its built output (onload="this.media='all'"). The restore matcher strips the onload attribute by name rather than by string value — a literal match would have silently no-oped in production, leaving tenant sites with a FOUC. Tests cover the source form, the Trunk-built/hashed form, and the entity-encoded form.

3.6 Asset Minification (esbuild)

Trunk does not minify CSS or JS output. A Lighthouse Minify CSS / Minify JavaScript audit flagged the unminified bundles. Added an esbuild 0.28.1 step to the deploy pipeline (baked into the runner image — no per-deploy download) that runs after all cargo steps (so a build.rs rerun can't clobber the output) and before precompression (so brotli/gzip wrap the minified bytes):

Asset	Raw	Minified	Brotli (min)
`shared_critical.css`	59.7 KB	36.8 KB	5.7 KB
`shared_deferred.css`	38.6 KB	29.8 KB	4.2 KB
`utilities.css`	29.9 KB	16.8 KB	3.0 KB
`components-*.css`	65.0 KB	45.5 KB	—
JS glue	76.1 KB	34.0 KB	8.1 KB

index.html is explicitly excluded (the  and  SSR marker comments must remain byte-exact for the presentation engine).

4. Mobile Web Vitals

4.1 Baseline and Diagnosis

A Lighthouse run on a Moto G Power emulation (Slow 4G) showed Perf 75, FCP 3.8 s, LCP 4.4 s, TBT 80 ms, CLS 0.055. The CSS @import waterfall was the dominant bottleneck — not the WASM.

After Track D (CSS fix), re-measuring confirmed:

	FCP	LCP	CLS	Render-blocking
Before	3.8 s	4.4 s	0.055	1,380 ms
After CSS fix	2.4 s	9.2 s	0.004	150 ms

The LCP regression to 9.2 s revealed the next layer: with the CSS path cleared, the WASM was now monopolising the network pipe on Slow 4G.

4.2 Hero Image Starved by the WASM

The LCP element is the hero logo (37.7 KB WebP) — present in the SSR HTML, on a preconnected CDN, with fetchpriority="high". Yet LCP trailed FCP by ~6.8 s on mobile. The diagnosis: Trunk injects a modulepreload for the WASM before the body <img> is discovered in the HTML stream. On Slow 4G (~1.6 Mbps capped), the WASM (1.27 MB brotli, ~6.4 s at that bandwidth) grabs the pipe first and starves the image until ~9 s. TBT was only 100 ms — confirming it was the network, not main-thread blocking.

Fix: preload the hero image in <head>, before the WASM modulepreload, with responsive media attributes mirroring the <picture> breakpoint:

<link rel="preload" as="image" type="image/webp" fetchpriority="high"
      href="https://cdn.mop.dev/mop_main/hero-logo-dark-mobile.webp"
      media="(max-width: 768px)">
<link rel="preload" as="image" type="image/webp"
      href="https://cdn.mop.dev/mop_main/hero-logo-dark.webp"
      media="(min-width: 769px)">

Only one variant is fetched (no double download); the debug SSR rewrite maps the CDN URL to /cdn for local development. Expected mobile LCP: ~9.2 s → ~3–4 s.

The durable fix is the WASM size reduction (§2) — a smaller WASM frees the pipe sooner regardless of the image preload order.

4.3 Hero Image Right-Sizing

hero-logo-{dark,light}-mobile.webp were 500×260 px, served into a ~211 px display slot. Resized to 440×229 via ffmpeg at WebP q80: ~10% smaller (43.3→38.6 KB, 42.7→38.3 KB) with the aspect ratio preserved to match the reserved <img width="440" height="229"> box — no new CLS.

The desktop full-res variant and the 10 KB navbar *-mini (also used on desktop at 60 px) were left as-is.

4.4 Back/Forward Cache Re-enabled

The SSR shell set Cache-Control: no-cache, no-store, must-revalidate on HTML responses. The no-store directive is the one that disables bfcache restoration on back/forward navigation — the browser cannot restore a page from the cache if it was marked no-store.

Fix: text/html responses now use no-cache, must-revalidate (bfcache-eligible). application/json keeps no-store — API responses may be user-scoped and a stale-cache regression in the fidelity POS was the original reason for no-store. Tenant pages keep their s-maxage opt-in branch, and the document shell carries no user data (auth hydrates client-side).

4.5 CLS Hardening and Animation Fixes

Beyond the root cause fix (§3.2 moving buttons into the critical CSS), several additional hardening measures:

The hero <img> width/height attributes updated from 364×189 to 440×229 to exactly match the new mobile intrinsic, reserving the precise LCP box.
prefers-reduced-motion: reduce in animations.css neutralizes the continuous non-composited loops Lighthouse flagged (copper shimmer's background-position, scrollPulse, float, pulse-glow) and pins [data-animate] reveals visible — a cleaner experience for motion-sensitive users and less main-thread work.
.btn transition: all replaced with an explicit property list (color, background, background-color, border-color, box-shadow, transform) so the GPU-composited transform hover-lift isn't dragged onto the main thread by the over-broad all transition.

5. Infrastructure and Developer Experience

5.1 Local Dev with CDN Assets

Before this work, a developer editing a shared CSS file had to deploy to cdn.mop.dev to see the effect — there was no way to test CDN-served assets locally. Added a #[cfg(debug_assertions)] branch in cdn_base_url() that returns the same-origin /cdn/{app} path (images, runtime locales, vendored libs), a matching debug rewrite in ssr_render_handler for the hardcoded CDN URLs in the HTML shell, and a Trunk [[proxy]] block forwarding /cdn → http://localhost:3000/cdn. Production is byte-for-byte unchanged — all three branches are cfg(debug_assertions)-gated.

5.2 Cross-Platform Build Script (No Shell Dependencies)

The CSS bundle generation needed to run on both Windows (developer machines, trunk serve) and Linux (CI runner). A Trunk.toml [[hooks]] approach breaks: sh is not on the Windows PATH and pwsh is not on the Linux runner. The solution is mop_frontend/build.rs — Cargo's build script runs cross-platform via cargo/trunk on both OSes with cargo:rerun-if-changed on each component CSS file ensuring regeneration on edit. No shell scripts, no Trunk hooks, no cross-platform maintenance burden.

5.3 Offline SQLx Cache Regeneration

The SQLx offline query cache (.sqlx/) was stale relative to the new batched-logging schema changes. Running cargo sqlx prepare regenerated 192 query type-checks, restoring SQLX_OFFLINE builds for CI and developer machines without a live database connection.

5.4 Cloudflare-Injected Scripts — Know What You Don't Own

Two Lighthouse findings that looked like our code turned out to be Cloudflare:

Deprecated Privacy Sandbox API warnings came from mop.dev/cdn-cgi/challenge-platform/…/main.js — the Bot Fight Mode / JS Detections probe script, injected by the edge. It was initially mis-attributed to the Cal.com embed; the embed removal fixed the __cf_bm cookie but not the deprecation warnings. Resolution: accept the diagnostic — the bot protection is worth more than a green Best Practices row.
email-decode.min.js in the critical-path chain is Cloudflare's Email Address Obfuscation, injected from /cdn-cgi/scripts/…/. Not ours to patch.
Cloudflare Fonts self-hosts our 4 Google font families under /cf-fonts/ and inlines the @font-face block into every HTML response, overriding our media="print" defer — but preserves font-display:swap. Pre-loading these fonts was explicitly rejected: on Slow 4G the font bytes would compete with the LCP image, re-creating the starvation §7.8's preload fixed.

Operator action taken: dropped JetBrains Mono from the Google Fonts request — --font-mono is only used in admin and article code blocks, never on the marketing critical path. One fewer @font-face family in Cloudflare's inlined block.

6. What Didn't Change (and Why)

Not every finding became a fix. A few deliberate non-changes worth noting:

Trunk CSS/JS minification disabled. Trunk's minify flag is global and runs minify-html over index.html, which strips comments — including the  and  markers that the presentation engine matches verbatim for white-label spoke injection. The byte saving is also near-zero after brotli. Instead, esbuild minification was added as a targeted post-build step that explicitly excludes index.html.
/auth/me deduplication deferred. The auth probes aren't naive: they re-verify is_admin server-side and hydrate cross-subdomain sessions. The safe improvement (a session-scoped cache behind a shared auth_me() helper) is a 7-file refactor touching login/admin/promoter/fidelity flows — high blast radius, deferred to a dedicated PR with those flows under test.
Full font self-hosting deferred. Render-blocking is already fixed (non-blocking load). Self-hosting removes the Google origin but means vendoring ~40 woff2 files with matching unicode-range @font-face blocks — a site-wide glyph-rendering risk if the subset structure is wrong. The lower-risk win (trimming unused weights) is still open.
serde-derive (~730 KB in WASM) — the dominant remaining dep cost. Only reducible by reducing the number of derived types or hand-writing deserializers — a correctness-critical refactor with diminishing returns relative to the effort.
Hero image re-compression declined. The logo is a detailed photographic image with an alpha channel, already encoded near its floor (q58 saves only ~640 bytes; dimension reduction would soften an already-sub-retina mark at the Moto G display size). Lighthouse's "21 KB savings" estimate assumed a flat graphic and was wrong.

7. By the Numbers

Category	Change	Result
Deploy time	~20 min → ~6 min	−70%
WASM raw	27.6 MB → 4.96 MB	−82%
WASM brotli	~1.5 MB → 1.01 MB	−33% from already-compressed baseline
Render-blocking budget	1,380 ms → 150 ms	−89%
CLS (homepage)	0.055 → 0.004	−93%
DB queries / tenant request	1 + new TCP → 0 (cached, 30 s TTL)	Eliminated
Request log inserts	80/s individual → bulk batched	−99% insert frequency
Critical-path JS (3P)	512 KB (unpkg)	Eliminated

All changes shipped with cargo clippy --all-targets -D warnings clean on both wasm32-unknown-unknown and native (--features ssr), and all backend/SSR/cache tests green.