The tell was the spreadsheet. A logistics operator had spent real money on an operations dashboard, and the people it was built for had quietly gone back to a shared spreadsheet, because the spreadsheet loaded and the dashboard did not.
How it started
The client did not ask for a performance audit. They asked whether they should rebuild the dashboard from scratch, having concluded the whole thing was rotten. Rebuilds are exciting and usually wrong as a first move, so we asked to measure before anyone reached for a clean slate.
Their own number was folklore — “it takes ages”. We replaced folklore with a p95: at the ninety-fifth percentile, the main view took roughly nine seconds before it showed anything a dispatcher could act on. Nine seconds, dozens of times a day, per person. The staff had done the maths with their feet.
What we found
The dashboard was not rotten. It was doing an enormous amount of correct work at exactly the wrong time.
Every load ran a fan-out of queries the moment the page opened — one per widget, several of them recomputing the same expensive aggregate over the same rows, none of them cached, all of them blocking the first paint. The front end then waited for the slowest one before rendering anything. One straggling query held the entire screen hostage.
There was no missing index conspiracy and no need for a bigger database. There was a sequencing problem wearing a performance problem’s clothes.
What we built
We changed when work happened, not what was allowed to run.
- The shared aggregate was computed once and cached, instead of independently per widget.
- The first paint was decoupled from the slow path: the screen renders the moment the cheap, common data is ready, and the heavier widgets fill in as they arrive rather than gating everything.
- The straggler query was moved off the critical path entirely and given a sensible loading state, so a slow report no longer freezes the fast ones.
- We added a small budget test to CI so a future well-meaning widget cannot quietly reintroduce the nine seconds.
No rebuild. The bones were fine; the choreography was not.
Results
p95 first-meaningful-load went from around nine seconds to under one. The change staff actually noticed was not the number — it was that they stopped emailing to ask whether the system was down, because a sub-second screen never looks broken.
The spreadsheet is gone. That was the real acceptance test, and it is worth more than a benchmark: people use the tool that respects their time, and now the fast tool and the correct tool are the same tool.