Getting Worse to Get Better
Some thoughts about improving in data science…and bowling, I guess.
Peter Licari 
It may come as a surprise to some people who know me, but I actually have other hobbies other than “doing stuff with data.” This post is concerns some experiences I’ve had with them. And, also, it’s about doing stuff with data. But before I talk about data, I want to briefly talk about bowling, golf, and running. (And then data, I promise).
Looking for a family-friendly activity sheltered from Florida’s triple-digit heat index, my wife and I recently introduced our daughter to bowling. For context, the last time I had gone bowling was our second date 13 years ago. I’m atrocious at bowling. My daughter, the 4 year old, beat me in the first round.1 A kindly gentleman named Pasquale gave me advice on my form. It made me much more likely to get the ball straight down the alley, but it was hard to keep all of these new biomechanics in mind. And while I got more strikes and spares than before, the awkwardness of the new form also led me to have more gutterballs. So my score more or less was the same as before.
1 She had bumpers and a ramp, but still!
I’ve been golfing a lot longer, nearly 25 years—though some years I’ll maybe only have the chance to play one or two rounds. I’m only just now consistently shooting below 100. After a round in May where I managed to shoot some surprisingly long shots (including one par 4 where my tee shot put me within 10 yards of the green), I tried gunning for the hole from more challenging positions rather than laying it up and playing it safe. Most of these attempts didn’t pay off. I shot my worst front 9 in nearly 5 years.
Running is something I’ve been doing almost as long as golf (20 years) but with much more consistency. Apart from writing, it’s been my most successful hobby. I’ve been trying to get myself back into 16-17 minute 5k shape. I recently breezed through a 10:55 2 mile time trial; almost exactly on pace for my goals. So I tried an aggressive hill workout to improve my stamina. My first interval of a planned 6 was right on pace and felt easy. But by the time I was done with the very next interval, I was doubled-over, gasping for air. It didn’t get any better. I not only missed my splits, I didn’t even finish the workout as planned. I tapped out after interval number 4. I wanted to kick myself in the ass, but I couldn’t even lift my legs enough to do it.
We’ve all heard that progress isn’t linear. That you’ll frequently hit plateaus before reaching new heights. But these weren’t plateaus, they were downturns. And these weren’t merely coincident with a recent improvement or a reversion to my mean; just a random walk around my baseline level of skill at these things. They were downturns caused by the fact I had recently improved. The better I got, the more ambitious I got. And ambition often begets errors.
That’s the frustrating, almost paradoxical thing: When you try to structurally elevate your performance, when you aim not just to score better on some metric but to be a categorically better performer, the immediate result is often a worse performance than if had you stuck with what you were doing before.
Let’s go ahead and tie this into data science now.
For better or worse, data science is a big-tent discipline. At its base is a chimerical mixture of statistics, engineering, and coding. This is then alloyed by whatever domains underlie the operation of the company writing your check (e.g., finance, law, marketing, psychology, chemistry, pharmacology, economics, user experience…) and whatever you need to effectively execute and communicate your work to others (design, IT, knowledge management, pedagogy, project management, people management…). It’s why there’s such high variance between different folks’ day-to-days; why some data scientists are doing cutting-edge LLM research, others are automating reports, and others still are working in Excel. All that is fine 2—but its far a from a uniform discipline.
2 Except that some folks are so heavily exposed to Excel, all my homies hate Excel.
3 For the happy majority who’ve never had to read an academic’s code or see how software engineers approach statistics, I should make clear that these aren’t exactly high bars.
There’s a joke from Josh Wills about the field that puts it more pithily: A data scientist is “[a person] who is better at statistics than any software engineer and better at software engineering than any statistician.”3 Even if you’re not in a start-up or operating as your office’s de facto “data person”, the discipline forces you to wear many hats. It’s impossible to master literally everything within the field’s purview. And because of its breadth, you have countless ways to improve yourself—and because of how yoked it is to tech’s “bleeding edge”, self-improvement is a professional prerogative. This improvement can come vertically (digging deeper into a known domain), horizontally (learning about an adjacent domain), or diagonally (likely picking up even more hats to wear).
No matter which route you take, though, I believe that taking data science seriously is to commit to continuous learning. To commit to continuous learning is to commit to being a continuous beginner. To commit to being a continuous beginner, for better or worse, is to commit to continuous mistakes.
As with all commitments, the commitment to continuous mistakes comes with some responsibility. Critically, it is not a carte blank check to fuck up. Indeed, a responsible commitment to being a perpetual beginner is to commit to enacting practices that constrain the frequency, severity, and extent of these mistakes. (After all, I may want to work on my fairway woods in golf but I should really balance that against the likelihood of accidentally slicing a ball through someone’s window).
It’s important to cater these practices to the potential threats you can anticipate:
- If you’re migrating processes into the cloud, you might want to set limits on how much money you can spend so you don’t silently wrack-up a huge bill.
- If you’re juggling identical values across different contexts in the same project, you can reduce the risk of copy-paste errors by placing these values in a single configuration file and then loading them in to your session.
- If you’re using a new statistical model or ML method, you can see if it will do what you expect it to with simulation—either with familiar data or with data you’ve synthesized yourself.
- If you’ve got a new data source, you can run tests after your transformations to ensure you’re getting the variables that you expect and that their values are sensible.
- If you’re working in a novel substantive domain, showing your results to subject matter experts will help ensure that you’re not proposing something that is highly unlikely or downright impossible.
I don’t want to make it sound like all these things are obvious. Some of them I’ve learned years ago, others I’ve learned within the last few weeks! And even those I’ve learned long ago, I’m constantly improving on how I go about practicing it. That’s part of why I said “the threats you can anticipate.” Embarking into new ways of doing things will expose you to novel “unknown unknowns” and you won’t be able to be proactive about threats you can’t yet anticipate. Even if you study-up and research this new domain (and you absolutely should do so as best as you reasonably can given the constraints you’re facing!), there are often subtle mistakes that will escape your best efforts until you’ve gained more experience. But I have noticed that, as you gain experience and expertise across a diverse range of practices, you’ll get better at anticipating potential mistakes. Not even just specific things to be looking out for, but broader types of concerns that you should at least consider preempting.4
4 For example, the cloud computing advice above isn’t something I learned from spinning-up stuff on AWS or Google or whatever—I learned it from generalizing an experience I had from when I started fielding surveys en mass.
5 Sometimes, this designing is iterative. You might make a similar mistake more than once if you’ve misapprehended the deeper source of the original error. Or, maybe, you understood it perfectly but are constrained in your ability to perfectly solve it. That’s fine! It’s going to happen! It becomes a problem when literally the exact same error is allowed to happen multiple times.
For all these reasons, and more, I’ve found that the most effective error-prevention strategies I’ve picked up—across all that I strive to do—is being earnestly open to constructive feedback and a dedicated practice of introspection. The feedback is imperative to help me learn what comes next and to catch any errors that slipped my notice. And the introspection is crucial when I reflect, research the error’s source more critically, and design systems to prevent it from happening again.5
I’m someone who takes a lot of pride in what I do.6 I don’t think of research as a job but as craft—and I like to think, after all the years I’ve put in, I’ve gotten pretty good at it. It’s that pride that inspires me to improve. But that pride and craftsman self-image are also double-edged swords. Because the errors that come with leveling-up my skills directly confront that pride and self-image. After all, we don’t think of dedicated craftsmen as making mistakes; their proficiency is a big part of what sets them apart from “mere” practitioners. As errors mount from trying something new, it’s easy to feel like you’re just an imposter.
6 As you could guess from my intro, that even extends to peripheral hobbies like bowling or pursuits central to my identity like running
7 Like when I, a music novice, attend concerts with my wife, who has played instruments nearly her whole life: She notices technical errors that go completely over my head.
It’s hard—but necessary—to remember that our observations of experts at work suffer from selection bias. We tend to see them performing at the top of their game. We tend to only see their end products and not what went into the production of them, or what hacky stuff they need to have happening behind the metaphorical curtain to make it work properly. And since their expertise is often distinct from our own, we may not even realize that mistakes are happening. As Annie Dillard writes in Pilgrim at Tinker Creek “I just don’t know what the lover knows; I just can’t see the artificial obvious that those in the know construct”.7
I’m writing this out explicitly because, again, I struggle with it myself: the mere existence of errors is not a negative commentary on your rigor. Provided that you’re doing your due diligence—you’re checking things over carefully and you’re not just haphazardly rushing something to “completion”—the errors don’t mean that you’ve regressed, that you’re failing, that you’re not being careful. If anything, catching errors from a system of checks can be evidence in support of your rigor.8 It’s certainly better than not catching mistakes.
8 Plus, you shouldn’t beat yourself up over the existence of more mundane errors anyways. You’re not a machine, nor should you aspire to be. To err is to be human. Again, as long as you’re being as diligent as you possibly can be, and only you can truly make that determination, then the best you can do is just…the best you can do!
9 I say more recently but do you know when that episode first aired? 2010! 14 fucking years ago! Jesus, I need to go lay down.
The physicist Neils Bohr once remarked “an expert is merely someone who has made every possible mistake in a narrow domain.” More recently, the character Jake the Dog from Adventure Time quipped: “Sucking at something is the first step at getting sort of good at something.”9
If you want to be proficient in several domains, which is almost required for broad fields like research and data science, you need to be prepared to make a lot of mistakes. And mistakes exhibit an interesting, weak ordinality: there are some mistakes that you can’t make until you’ve made others first. You can’t make expert mistakes until you stop making beginner mistakes. And many pursuits exhibit opportunity costs to endeavoring on the expert route. A high-risk, high-reward workout that explodes in your face comes at the expense of a solid one that maintains your current state. An ambitious tee shot that lands in the water costs you strokes compared to your usual “lay it up and play it safe” approach. At least until you’ve done the workout so often that it’s no longer “high risk”; and at least until you’ve played the hole enough that the tee shot is no longer ambitious.
Once you’ve hit a certain point, the only way to get better is, for a time at least, to get at least a little worse.
But only a little. And only for a time.
I’d like to be able to conclude this by giving it some typical, self-help fairytale ending: “now I bowl over 200”; “now I shoot under par”; “now I win every road race I enter”; “now every analysis I make is worth its weight in gold”.10 But it’d be bullshit. I am noticing improvements though: even in the things I would call myself an expert in (which definitely isn’t golf or bowling).
10 Considering the weight of a bit and the weight of a word, though, this one may actually be true.
So I can say I’m making progress. And since we are all always works in progress, that seems good for now. I’m getting worse. I’m getting better.
I hope you are too.
Reuse
Citation
@online{licari2024,
author = {Licari, Peter},
title = {Getting {Worse} to {Get} {Better}},
date = {2024-06-20},
url = {https://www.peterlicari.com/posts/getting_worse_getting_better_2024/},
langid = {en}
}