This series was never about learning three frameworks. It was about changing how you engage with the causal claims that flow through your work every day: the ones you write, the ones you review, and the ones you use to justify decisions that affect real people and real budgets.
Theory is easy to nod along to. Application is harder. So this time: no new concepts. Instead, a decision framework you can use the next time you write, review, or commission an analysis that makes a causal claim.
Not every question requires the same approach. The right lens depends on what you’re trying to do.
Situation 1: “Did the program work?”
You have a program that’s been implemented. You have data on participants and outcomes. You need to know whether the program caused the outcomes you’re observing.
This is the classic causal inference problem. The core question is: what would have happened without the program? That’s your counterfactual, and your entire analysis is only as good as your strategy for constructing it.
Before you run anything, ask:
What is my counterfactual? If it’s a before/after comparison, you’re assuming nothing else changed. If it’s a comparison of participants to non-participants, you’re assuming the two groups would have had the same outcomes absent the program. Both are strong assumptions. Can you defend them?
What am I controlling for, and why? Draw the causal relationships between your treatment, your outcome, and every variable you’re considering as a control. Is each one a confounder, a mediator, or a collider? If you can’t answer this, your specification is hiding assumptions, not eliminating them.
Is my estimate an average? If so, who is it averaging over? Could the effect be positive for some and negative for others? Does the average apply to the population leadership cares about, or just to the people who happened to be in your sample?
Situation 2: “Is our analysis identifying what we think it is?”
You have an analysis, maybe one you’ve inherited, maybe one a contractor delivered. It makes a causal claim. You need to assess whether that claim is credible.
This is an assumptions audit. The question isn’t whether the statistics are correct, it’s whether the analytical design supports the causal interpretation being drawn from it.
Ask:
Can I draw the causal model? Take the analysis and draw the relationships it assumes between variables. If the authors haven’t made their assumptions explicit, make them explicit yourself. Where do you agree? Where are you uncertain? Where do you think they’re wrong?
What’s the identification strategy? How is the analysis generating variation in the treatment that’s unrelated to the outcome? If it’s “we controlled for observables,” that’s not an identification strategy, it’s a hope that nothing unobserved matters. If it’s a natural experiment, an instrumental variable, or a discontinuity, assess whether the assumptions behind that strategy hold in your context.
Could I break this result? What would have to be true for the causal claim to be wrong? Is there a plausible confounder the authors haven’t addressed? A selection mechanism they’ve ignored? If you can easily construct a story where the result disappears, the analysis hasn’t done enough to rule it out.
Situation 3: “What would happen if we did something new?”
Leadership wants to know the likely effect of a policy that hasn’t been tried, in a population that hasn’t been studied. There’s no experiment to run and no natural experiment to find.
This is the hardest situation, and the one where most people either avoid (“we’d need more data”) or overreach (“based on a similar program…”). Neither is adequate.
Ask:
Do we understand the mechanisms? If a similar program worked elsewhere, why did it work? Through what channels? If you can’t articulate the mechanisms, you can’t predict whether the effect will transfer to a new context. “It worked in City A” is not evidence it will work in City B unless you understand what about it worked and whether those conditions hold in City B.
Who are the decision-makers, and what are they choosing? The people your policy targets aren’t passive. They’ll respond to incentives, constraints, and information. How will they make decisions about participation, effort, or compliance? How does that affect who benefits and by how much? If your prediction doesn’t account for the behaviour of the people in the system, it’s a mechanical projection, not a causal analysis.
What assumptions am I making, and can I state them? Every prediction about an untried policy rests on a model, implicit or explicit. Make yours explicit. Write down the mechanisms you believe are operating, the behavioural responses you’re assuming, and the conditions you think are necessary for the effect to materialise. Then ask: which of these could I be wrong about, and how much would it matter?
Using this in practice
These three situations aren’t mutually exclusive. A single project might start in Situation 1 (did our pilot work?), move through Situation 2 (is the evaluation credible?), and end in Situation 3 (should we scale it?). The framework isn’t a flowchart, it’s a set of habits.
The common thread across all three is the same discipline this series has argued for from the beginning: make your assumptions explicit. Whether you’re constructing a counterfactual, auditing someone else’s analysis, or predicting the effect of something new, the quality of your causal reasoning depends on your willingness to state what you believe about how the world works, and to be honest about what you don’t know.
The regression coefficient by itself is not a causal effect. The counterfactual is never observed. Your diagram reveals whether your controls help or hurt. And mechanisms are what let you generalise beyond the data you have. Use them.
Sources for going deeper:
- Angrist, J. & Pischke, J. (2014). Mastering ‘Metrics. Princeton University Press. The most accessible entry point for Rubin-style causal inference.
- Angrist, J. & Pischke, J. (2014). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press. The second most accessible entry point for Rubin-style causal inference.
- Pearl, J. & Mackenzie, D. (2018). The Book of Why. Basic Books. DAGs and causal reasoning for a general audience.
- Huntington-Klein, N. (2021). The Effect. Chapman & Hall/CRC. Bridges Rubin and Pearl, excellent on identification. Free at theeffectbook.net.
- Heckman, J. & Pinto, R. (2022). The Econometric Model for Causal Policy Analysis. Annual Review of Economics, 14, 893–923. The case for structural models and mechanisms in policy evaluation.
