Ask most actuaries where their models live and the honest answer is often "a very large, very fragile Excel workbook." That worked for decades. It does not work under principle-based reserving, where the computational demands have exploded. I've built a VM-21 calculation engine and a suite of cashflow, hedging, and risk-measurement models on Databricks and risk tools, and the performance difference against a legacy Excel process is not incremental — it's transformational. Here's why the shift matters and how to do it responsibly.

Why the old approach breaks

Principle-based reserving is computationally brutal by design. The VM-21 stochastic reserve is calculated over 10,000 economic scenarios generated by the American Academy of Actuaries' Interest Rate Generator. For pricing and asset-adequacy work you often need nested stochastic projections — scenarios inside scenarios — and, as RNA Analytics notes, "the demand for nested calculations is growing, with the accompanying computational burden expanding exponentially."

To put numbers on it, Milliman's oft-cited illustration is sobering: a 1,000-scenario model with reserves based on 1,000 paths at each valuation point, run over a 30-year monthly projection, requires each policy's cash flows to be projected 360 million times. That is simply not something an Excel workbook — or even a spreadsheet farm — handles gracefully. Firms have historically thrown hardware at it, running projections on grids of as many as 1,500 PCs.

The problem isn't just runtime. It's fragility, lack of version control, poor auditability, and the enormous key-person risk of a spreadsheet only one person truly understands.

What Databricks changes

Moving these calculations onto Databricks and PySpark attacks the problem on several fronts at once. Spark distributes the computation across a cluster, so the embarrassingly parallel structure of a stochastic projection (each scenario is independent) maps naturally onto the platform. Delta Lake gives you versioned, transactional data with full lineage — so you can reproduce any run exactly, which is precisely what auditors and validators want.

The industry results are striking. Per Microsoft's customer story on Milliman's cloud HPC work, "insurers now complete complex actuarial runs in hours instead of days, with failure rates below 1% and up to 70% faster runtimes." One large insurer evaluated 7.2 million risk scenarios — the equivalent of 214 years of computation — in just three days with zero failures. WTW's RiskAgility FM claims run-times roughly 10x faster than legacy software. My own experience building a VM-21 engine on Databricks/PySpark echoes this: materially faster runs than the Excel process it replaced, with the added benefit of a controlled, reproducible environment.

There's a cost dimension too. In a cloud environment, faster runtimes translate directly into lower spend, and workload-specific cluster tuning lets you match compute to the job. Milliman reported one major UK client growing actuarial run volume by 20% year over year while overall compute costs fell 20% over the same period.

The part people skip: model risk

Here's where I want to be emphatic. Speed is seductive, but a fast model that nobody has validated is a liability, not an asset. When I build these engines, I formally assess model appropriateness, limitations, and model risks — because a reserving calculation is a regulatory artifact, not just a number.

That means documenting assumptions and their justification; testing the Spark implementation against the legacy model to confirm the answers reconcile before you trust the new one; understanding where approximations (scenario reduction, policy clustering) are used and quantifying their impact; and building governance and sign-off into the pipeline. The NAIC's tightening expectations on model governance make this non-negotiable, and the American Academy of Actuaries is explicit that PBR increases modeling complexity "while at the same time requir[ing] more transparency for governance." Under PBR, the model is the reserve, and regulators expect transparency to match the increased complexity.

Cloud platforms actually make governance easier if you use them well — version control, lineage tracking, and reproducibility are built in rather than bolted on. That's a genuine upgrade over the "which version of the spreadsheet is the real one" problem.

How to approach the migration

My advice to clients is to treat it as a staged transition, not a big-bang rewrite. Start by reconciling a single well-understood block against the legacy model until you trust the new engine completely. Then expand scope, build the governance and documentation layer in parallel, and only decommission the legacy process once you have demonstrated equivalence and stability. Model-efficiency techniques like policy clustering and scenario reduction can cut runtime and cost further — but only once you've quantified their impact on the answer. The efficiency gains are real, but they're only worth having if the answers are right and defensible.

If you're facing VM-21, VM-22, or broader PBR computational demands and your current process is straining, this is exactly the kind of modernization I help firms execute — fast and validated. Let's talk.