I usually want my prediction models to be wrong

I generally hope that my time series models wind up wrong.

I don’t just mean in the flippant sense of “all models are wrong but some are useful”. I mean literally wrong. As in my prediction didn’t come to fruition.

One of the fun things about doing social science is that your subjects react to your analysis. It’s kind of funny because, like, IID goes out the window for future work if your work gets popular. By observing the world, we seek to influence its evolution.

9 times out of 10, I make forecasts about sociological systems. If things remain constant, we can reliably get within 10-15% mean absolute percentage errors. Sometimes less. And that’s when we try to avoid black-boxes and atheoretical modeling choices; we ultimately also have to provide interpretations to stakeholders. If we didn’t have that constraint (though I’m honestly glad that we do) then we could probably get closer.

So it’s not that I can’t do a decent job modeling this stuff. It’s just that I make these predictions in the hopes that people with the power to change (at least some) of the inputs heed what I’m saying and try to make things better. Even if, qualitatively, my results jump off the page and have a parade because of how awesome the implications are for the business, things can always be improved. (If for no other reason than the fact that it will eventually revert to the mean and we’ll then be glad to have instituted improvements so that we’re not as low as we could’ve been).

Of course, there’s a bit of grim satisfaction when things don’t change and we hit what we, in fact, predicted. But no amount of internal “I told them so” can douse the fact that it’s a Pyrrhic victory.

I want to be right in a counterfactual sort of way. I want my predictions to be the reality we would observe if we were ignored. I want it to be outperformed by what actually happens because stakeholders took our predictions seriously and worked to make things better. So I want to be wrong in that I want to be proven wrong.