A minister asks: if we expand the childcare subsidy, how many more parents will take up work?
We look at the data. Parents who use subsidised childcare work more hours than parents who don’t. The relationship is strong, statistically significant, and stable across states. Good enough to answer the minister?
No. And the reason it isn’t good enough is the whole reason causal reasoning matters for policy design.
Over the past five weeks, we’ve been talking about what it takes to make a credible causal claim. But here’s something worth stepping back on: causation isn’t always the question.
“How many graduates will enter the workforce next year?” is not a causal question. It’s a request for a number, and the number needs to be accurate. “Did the tuition subsidy increase graduation rates?” is a causal question. It requires a counterfactual, an identification strategy, everything we’ve discussed. “What would happen if the subsidy were redesigned?” is different again: forward-looking, but in a changing policy environment where past patterns can’t be extrapolated.
If you work with large administrative datasets or run simulation models, you’ve probably lost hours to slow code. Usually the fix is obvious in hindsight, but only once you know which kind of slow you’re dealing with.
Most analytical work falls into one of two categories.
This series was never about learning three frameworks. It was about changing how you engage with the causal claims that flow through your work every day: the ones you write, the ones you review, and the ones you use to justify decisions that affect real people and real budgets.
Theory is easy to nod along to. Application is harder. So this time: no new concepts. Instead, a decision framework you can use the next time you write, review, or commission an analysis that makes a causal claim.
Not every question requires the same approach. The right lens depends on what you’re trying to do.
Over the past three weeks we’ve built a toolkit. Rubin gave us the language of potential outcomes and the counterfactual. Pearl gave us DAGs to map our assumptions and check whether our analysis can identify a causal effect. Both frameworks assume you have data — either from an experiment or from observational data with a credible identification strategy.
But what happens when you don’t? When leadership asks “what would happen if we implemented policy X?” and policy X has never been tried? There’s no treatment group. No control group. No natural experiment. No data to construct a counterfactual from.
Continue reading →We’ve established that causal inference requires a counterfactual, and that every analytical strategy is a way of constructing a credible stand-in for what we can’t observe. But we haven’t addressed a more basic question: how do you know whether your specific analysis is actually estimating a causal effect?
If your regression coefficient is not a causal effect, what is a causal effect?
Here’s the answer, and it’s deceptively simple.
The causal effect of an additional year of education on a person’s earnings is the difference between what they earn with that year of education and what they would have earned without it. That’s it. Not the difference between educated and uneducated people. The difference between two states of the world for the same person, one that happened and one that didn’t.
Quick thought experiment. You regress earnings on years of education and get a positive coefficient. More education → higher earnings. Done. Done?
Now add parental income as a control. The coefficient on education shrinks. Add a measure of cognitive ability. Shrinks again. Add motivation, grit, neighbourhood quality. It keeps moving.
So which coefficient is the “real” effect of education?