Calibrating MMM with experiments: ROAS as a parameter

The cleanest way to feed an incrementality experiment into a media mix model is to make the ROAS it measures a model parameter and drop the experiment value straight into its prior. Google's Media Mix Model Calibration With Bayesian Priors^[1], from January 2024, proposes this reparameterization. In simulation the calibrated model's posterior ROAS standard deviation fell to about one fifth of the uncalibrated model's^[1]. The evidence is simulation, and when the confounder is unobserved the bias does not fully disappear.

This article is based on the research.google PDF (January 2024, seven authors)^[1], read in full with the numbers checked against the source. The authors note up front that the causal reading rests on the uncheckable assumption that all confounders are controlled. We run multi-step measurement pipelines ourselves, so we have an interest in this topic.

The problem: why experiment results are hard to feed into an MMM

A marketing mix model and an incrementality experiment measure different faces of the same ad effect. An MMM uses aggregated history to measure each channel's long-term average efficiency on top of baseline sales, and loses precision on small channels with little spend variation^[1]. An experiment randomly splits exposed and withheld groups to measure one channel's short-term causal effect at one point in time. One looks wide and long, the other narrow and short.

The strength of a Bayesian MMM is that it can absorb the experiment result as a prior^[2]. The trouble is the process of doing so. The conventional method iterates over prior distributions for the response coefficient ${\beta }_m$ until the implied ROAS prior matches the experiment^[1]. Two things bite. ${\beta }_m$ is entangled with the saturation and carryover parameters, so it must be tuned analytically, which is slow and takes expertise. And if the desired ROAS prior has a small standard deviation, no independent ${\beta }_m$ prior may produce it, because ${\beta }_m$ is not the only contributor to ROAS variance.

The marketing mix model and the definition of ROAS

The base model is a Geo-level Bayesian Hierarchical MMM (GBHMMM)^[2]. Sales $y_{t,g}$ for geo $g$ at time $t$ is a sum of a baseline term, media terms, and control terms.

Here $x_{t,m,g}^{\ast }$ is the spend series for channel $m$, $z$ is a control variable, and ${\tau }_g$ is the geo baseline. Saturation and carryover enter through two transforms.

The Hill function captures diminishing returns as spend grows, with $K$ the half-saturation point. Adstock captures carryover through a retention rate that decays past spend over up to $L$ periods, where the retention rate lies between zero and one^[2].

ROAS is incremental sales per dollar of media spend. The authors define it in the Rubin causal (potential outcomes) framework: sales in the world where the channel is on, minus the world where it is off, divided by spend^[1]. Written through the model, ROAS is

where $C_{t,m,g}$ is spend and $F$ is the transform difference between the on and off worlds. The key point is that ROAS is a function of several parameters, ${\beta }_{m,g}$ among them. There was no handle to reach ROAS directly inside the model.

The core idea: reparameterize so ROAS is a model parameter

The method rebuilds the model so ROAS itself becomes a parameter. Write ${\beta }_{m,g}={\beta }_m+{\eta }_m{Z}_{m,g}$, then solve the ROAS equation for ${\beta }_m$, and ${\beta }_m$ becomes a closed-form function of ROAS.

Because spend $C_{t,m,g}$ and the window $[T_0,T_1]$ are known, $H$ is computable in closed form. Substituting $H$ in place of ${\beta }_m$ in the original model gives a model parameterized by ${\mathrm{ROAS}}_m$ instead of ${\beta }_m$. Where the old model placed a prior on ${\beta }_m$, the new one places it on ${\mathrm{ROAS}}_m$. The range an experiment reports (mean, confidence interval) maps onto a log-normal ROAS prior, which prevents negative values, and drops straight in^[1].

The framework also handles two awkward cases. When the experiment measures only a short-term effect that diverges from the modeling-window ROAS, you can narrow the calibration window to the experiment period or widen it to the full window^[1]. When a single channel has several experiments, the per-experiment ROAS values are not identifiable (many combinations give the same ${\beta }_m$), so the authors recommend collapsing them into one spend-weighted aggregate ROAS^[1]. The same reparameterization applies to BMMM^[3] and any Bayesian MMM with a one-to-one map between ROAS and one parameter.

Setup and results

The evidence is a simulation: one media channel, one control variable, 100 geos, and 104 weeks (two years) generated from the model^[1]. The control variable mimics seasonality with a sinusoid plus noise, and media spend is made positively correlated with it to plant a confounding structure. Calibration compares a baseline model with non-informative priors against a calibrated model with a ROAS prior.

The headline number is uncertainty. The uncalibrated model's posterior ROAS standard deviation was about five times the calibrated model's^[1]. Both the full-window and the sub-window (quarter 3) priors tightened the posterior to a similar degree, with the full-window one slightly narrower.

Bias is more careful. When the confounder is unobserved and measured only through a noisy proxy, ROAS estimates were biased whether calibrated or not^[1]. The ROAS prior reduced that bias substantially, and the full-window prior reduced it more than the short-term sub-window prior. The authors state plainly that this simulation is an ideal case free of omitted variables and model misspecification.

Limits

All of the paper's evidence is simulation, with no validation on real advertiser data^[1]. The authors name several limits of their own.

• The causal reading rests on conditional exchangeability, an uncheckable assumption. You must control all confounders, and controlling a mediator or collider instead can introduce bias.
• When the confounder is unobserved, calibration cannot remove the bias entirely. It only reduces it.
• The ROAS prior must sit near the truth. If a short-term experiment over- or under-states the long-term effect, you are injecting a biased anchor.
• Several experiments on one channel are not individually identifiable, so they must be collapsed into one aggregate ROAS.

So the method cleans the channel for feeding experiments into a model. It does not stand in for the quality of the experiment or the control of confounders.

What we take from it at TRAIL

What stays with me from this paper is not the five-times figure but how it was reached. The point is simple: do not bury the quantity you want to measure deep in the model and grope at it through a proxy. Parameterize the quantity you measure. ROAS is what the business cares about and what the experiment measures, yet the model was written in terms of a coefficient ${\beta }_m$ nobody has intuition about. So injecting experiment knowledge meant groping at the ${\beta }_m$ prior. Make ROAS a parameter, and the experiment value walks straight in.

I work on measuring AI search visibility, so this structure does not feel foreign. The visibility scores and share-of-voice numbers a GEO dashboard shows are proxies, tangled with competitor activity, model updates, and query mix. There is no slot in them to drop the result of a causal experiment, the kind that says "we changed this page like this and citations went up." Ad measurement is already at the point of feeding experiments in as priors, while measuring AI citations is still stuck on proxy indices.

So the measurement we build is not one dashboard number but a loop: track a prompt set across engines on a schedule, measure what we changed before and after like an experiment, and let that result become the baseline for the next round. This paper shows what that loop looks like once it matures in ad measurement. The split between component scores and in-loop outcomes is the same one seen in content entering the retrieval pool but not being cited and in isolated accuracy failing to predict agent-loop success. How you even count a citation is already a measurement decision, which the measurement definitions of generative engine optimization take up. In any domain, the number you can trust is the one measured inside your own loop, by changing something and re-measuring.

The current design of incrementality experiments runs through Google's trimmed match geo experiments^[4], and the view of measurement as a new standard for digital advertising runs through HBR's 2023 piece.