A minister asks: if we expand the childcare subsidy, how many more parents will take up work?
We look at the data. Parents who use subsidised childcare work more hours than parents who don’t. The relationship is strong, statistically significant, and stable across states. Good enough to answer the minister?
No. And the reason it isn’t good enough is the whole reason causal reasoning matters for policy design.
The minister is asking a causal question, whether she uses that word or not. She wants to know what would happen if we changed the policy. That is a comparison between a world where the subsidy is expanded and a world where it isn’t. Neither of those worlds is what we observe in the data. What we observe is a mix of parents who chose to use the subsidy and parents who didn’t, and the choice is not random. Parents who use subsidised childcare tend to be the ones who were already planning to work more. They sought out the subsidy because it fits a plan they already had.
So the correlation in the data is telling us two things at once: something about the subsidy, and something about who chose to use it. We cannot separate them by looking harder at the same data. And the two answers point in different directions for policy.
If most of the correlation is selection, that is, parents who wanted to work took up the subsidy, then expanding it to everyone will do less than the correlation suggests. Many of the newly eligible parents won’t take it up, and many who do would have worked anyway. The effect of the policy is smaller than the data implies.
If most of the correlation is causal, that is, the subsidy made work feasible for parents who wouldn’t otherwise have taken it up, then expansion could do a lot. New parents come into the scheme who really do change their behaviour because of it.
Same correlation. Two very different policies. The decision the minister has to make depends on which world we’re in, and the correlation on its own cannot tell her.
This is what causal reasoning is for. It’s not a methodological preference or a purity test. It’s the discipline of separating “what we see” from “what would happen if we changed something”. Policy design is always the second question. The correlation is at best a starting point for it, and at worst a confident answer to a different question altogether.
None of this requires anyone to abandon existing data or existing analysis. It requires being clear about the question. Before we ask what the data says, we need to ask what decision the analysis is for, and what comparison that decision requires. That comparison almost always requires causal analysis. Once we’ve named it, we can be honest about how close our data gets us to it, and where the gaps are.
Everything we’ve just worked through with the subsidy comes down to three concerns. What is the correlation being compared against. What else might explain it. And who ended up on each side of the comparison in the first place. Stated as questions, these are below.
Compared to what?
Every causal claim is a comparison between what happened and what would have happened otherwise. The “otherwise” is never observed, which is what makes causal reasoning hard. When someone says the program worked, the immediate follow-up is: worked compared to what? No program? A cheaper program? A different program the same people would have accessed instead? The decision-maker is not choosing between an option and a void. She’s choosing between two active options, and the relevant comparison is between them. A number without a named comparison is not yet an answer.
What else was going on?
Two things can move together because one causes the other, or because a third thing causes both. Parents who use subsidised childcare work more. They also tend to have partners in stable employment, live in areas with better transport, and have already decided to be in the workforce. Any of those could be doing the work we’re attributing to the subsidy. The question is asking what else changed at the same time, or what else distinguishes the groups being compared. In policy settings the answer is almost always “a lot”, and the observed difference is a sum of the policy effect and everything else. Causal design is the work of ruling those alternatives out, or bounding how much they could plausibly matter.
Who ended up in that group, and why?
Even with the comparison specified and the confounders addressed, we still have to ask how people came to be in the groups we’re comparing. If they sorted themselves, and the sorting relates to the outcome, the comparison is contaminated. Parents who use the subsidy are not a random slice of eligible parents. They’re the ones for whom it fit an existing plan. This matters twice for policy design: it changes what we think the current scheme does, and it changes what we should expect from expansion, because expansion reaches people who didn’t select in under the current rules. The effect on them is not the effect measured on the ones who did.
These three questions cover the main routes by which a correlation fails to answer a causal question: the wrong contrast, an omitted common cause, or non-random sorting into groups. An analyst who can answer all three convincingly has a causal claim. An analyst who can’t answer any of them has a correlation dressed up as one.
Most analytical failures in policy design are not failures of technique. They are failures of question being answered. The correlation is answered correctly. It just isn’t the answer to what anyone needed to know.
