If your regression coefficient is not a causal effect, what is a causal effect?
Here’s the answer, and it’s deceptively simple.
The causal effect of an additional year of education on a person’s earnings is the difference between what they earn with that year of education and what they would have earned without it. That’s it. Not the difference between educated and uneducated people. The difference between two states of the world for the same person, one that happened and one that didn’t.
This is the counterfactual. And the uncomfortable part is that you can never observe it.
The fundamental problem of causal inference
This idea was formalised by the statistician Donald Rubin in the 1970s, and it’s now known as the Rubin Causal Model or the potential outcomes framework. The setup is straightforward: for each person, there are two potential outcomes. The outcome if they receive the treatment, call it Y(1), and the outcome if they don’t, Y(0). The causal effect is Y(1) – Y(0).
The problem: you only ever observe one of these. A person either got the additional year of education or they didn’t. You never see both. The unobserved outcome, the road not taken, is the counterfactual, and it’s fundamentally missing data.
This isn’t a statistical problem you can solve with a bigger sample or a better model. It’s a logical constraint. No dataset in the world contains both potential outcomes for the same person. Every method for estimating causal effects, every single one, is ultimately a strategy for constructing a credible stand-in for the missing counterfactual.
How causal inference becomes possible anyway
If you can never observe the individual causal effect, how does anyone make causal claims at all?
The move that makes causal inference possible is a trade-off: you give up on individual effects and aim for the average treatment effect, the ATE. Instead of asking “what would this person have earned without the extra year of education?” you ask “what would people like this person have earned, on average?”
In a well-designed randomised experiment, this works because random assignment ensures that the treatment and control groups are, on average, identical in every respect except the treatment itself. The average outcome in the control group is a credible estimate of the counterfactual for the treatment group. The difference between the two averages is the ATE.
This is powerful, but it comes at a cost your analysts should recognise. The ATE is an average. It tells you that the program improved outcomes by X% on average across the group. It doesn’t tell you that every participant benefited by X%. Some may have gained far more, some may have gained nothing, and some may have been made worse off. The average conceals all of that.
When you write “the program improved employment outcomes by 8 percentage points,” you’re reporting an ATE. That’s a legitimate and often useful quantity. But it’s worth pausing on what it doesn’t tell you, because leadership will often read it as “the program works,” when what it actually means is “the program worked on average, for this population, under these conditions.” We’ll return to what the average hides later in this series.
Why this changes how you think
Once you see causal inference as a missing data problem, things that seemed like technical choices reveal themselves as assumptions about the counterfactual.
When you run a regression of earnings on education “controlling for” ability, you’re implicitly assuming that people with the same measured ability but different education levels are valid stand-ins for each other’s counterfactuals. That’s a strong claim. Is someone who scored the same on a test but chose not to pursue more education really a good proxy for what the educated person would have earned without that education? Maybe. Maybe not. The regression doesn’t tell you, it just runs either way.
When researchers used compulsory schooling laws as a natural experiment (as we discussed last week), they were making a different and more transparent counterfactual argument: people born just before and just after a school-entry cutoff are essentially the same, except one group was compelled to stay in school slightly longer. The counterfactual is the other side of an arbitrary birthday cutoff, not a regression-adjusted comparison group chosen by the analyst.
Both approaches construct a counterfactual. The difference is in how credible that construction is and how explicitly the assumptions are stated.
The test for your own work
Next time you write or review an analysis that makes a causal claim, “program X improved outcome Y,” ask a simple question: compared to what?
What is the counterfactual? What would have happened to these participants without the program? And how do you know?
If the answer is “we compared outcomes before and after the program,” you’ve assumed that nothing else changed, no economic shifts, no policy changes, no maturation effects. That’s your counterfactual, and it’s often a weak one.
If the answer is “we compared participants to non-participants,” you’ve assumed that the two groups would have had the same outcomes absent the program. But people who enrol in programs are usually different from those who don’t, more motivated, more in need, or more connected. That selection is exactly the problem Rubin’s framework forces you to confront.
The framework doesn’t give you the counterfactual. Nothing can. What it gives you is the discipline to ask whether your stand-in for the counterfactual is defensible, and to be honest when it isn’t.
Pushback and disagreements welcome. That’s the point.
Further reading:
- Holland, P. (1986). Statistics and Causal Inference. Journal of the American Statistical Association, 81(396), 945–960. https://doi.org/10.2307/2289064.
- Imbens, G. & Rubin, D. (2015). Causal Inference for Statistics, Social, and Biomedical Science: An Introduction. Cambridge University Press.
- Angrist, J. & Pischke, J. (2015). Mastering ‘Metrics. Princeton University Press. Ch. 1–2 cover potential outcomes and regression in the most accessible way available.
