Why Jonah Lehrer's Criticism of Sabermetrics is so Disappointing
Let me say this up front: I like Jonah Lehrer. I am a fan of his writing and generally find him quite informative and thoughtful. This probably explains my visceral reaction to his latest, a dismissive column at Grantland that criticizes the use of analytics in various sports, including baseball.
As JD mentioned, it's a bad column and disappointing. I don't simply claim it to be bad because I disagree with his conclusions. I say it's bad because it is sloppy and Lehrer is a smarter guy than this.
The argument in short:
My worry is that sports teams are starting to suffer from a version of the horsepower mistake. Like a confused car shopper, they are seeking out the safety of math, trying to make extremely complicated personnel decisions by fixating on statistics. Instead of accepting the inherent mystery of athletic talent — or at least taking those intangibles into account — they are pretending that the numbers explain everything. And so we end up with teams that are like the worst kind of car. They look good on paper — so much horsepower! — but they fail to satisfy. The dashboard is ugly, the frame squeaks, and the front seats make our ass hurt.
First, there isn't a single team nor analyst that claims that sabermetrics capture all there is to know about athletic talent and performance. Teams use a variety of other metrics and method to assess player talent as well as likelihood to succeed.
Second, and more important, Lehrer's main argument shouldn't be that teams are assembling bad teams because of a narrow-minded focus on things they can quantify. The argument should be that teams that don't think deeply about what are the right metrics and how much variance they account for in player achievement will fail just as much as those teams that used to generally ignore analytical approaches to the game.
Data and statistics are not to blame for bad decisions--their misapplication is. Lehrer is trying to get at this point, but his misinformed broadside against sports analytics makes it real easy to miss.
Moreover, Lehrer's examples are just plain bad.
One example he uses is the NBA Champion Dallas Mavericks. He claims that because the Mavs did not match up statistically against each of their playoff opponents, yet prevailed, shows that a reliance on these statistics is flawed.
Lehrer is both right and wrong.
He's right in that relying solely on these statistics and not taking into account context would be a mistake. Again, I am waiting for someone to raise their hand and say they advocate this, particularly on a game by game or series by series basis.
He's wrong in that this example in no way proves or disproves the usefulness or accuracy of the underlying analytics. The playoffs are a handful of 7-game series. Anyone with a basic understanding of probability knows this is the last place you would look to prove or disprove the effectiveness of such measures given the role of randomness and luck.
Finally, Lehrer continually claims that a long list of intangibles largely determines the outcomes of games. He goes so far as to say that:
not everything that matters can be measured, and that success in sports (not to mention car shopping) is shaped by a long list of intangibles.
Now, he may very well be right, particularly if we are trying to make point predictions. But my first question is, if it can't be measured how do we know that it affects outcomes they way he claims? It's like someone who sees a UFO jumping to the conclusion that since we can't explain what it is it must be aliens. If it's unidentified you have no justification for making that leap. In Lehrer's case, neither does he.
Furthermore, sabermetrics--and any analytics whose subject is people and not atoms--is all about context and probabilities. When players don't perform as expected it could very well be for "intangible" reasons. This doesn't negate sabermetrics, not one bit. It goes hand in hand.
Like with any technology, sabermetrics and other analytical approaches won't by themselves lead to optimal decision making. Their is a mindset that must accompany them. That's a message worth delivering. Unfortunately, Lehrer is the wrong messenger. At least, he was on this day.
27 comments
|
8 recs |
Do you like this story?
Comments
So happy you wrote this, Bill.
Made a comment on JD’s FanShot after reading Lehrer’s article, and I was just really disappointed by the entire thing. Clearly, the mainstream still hasn’t grasped what sabermetrics is about- figuring out where value lies, and what contributes to winning, not trying to quantify entire sporting events with a series of numbers- and that makes me want to tear my hair out as a saber guy.
Here’s the comment I made before:
I always feel uncomfortable when mainstream writers make the assumption that anyone who loves sabermetrics ignores the existence of unquantifiable variables. The larger point is that we should put more value into things that we can quantify, but I don’t know why the next logical step in that reasoning should be that unquantifiable variables don’t exist.
Not to mention that he’s ignoring the massive idea that the team with less talent does often win the game. Sure, the Mavs won the championship over the statistically superior Heat. That doesn’t mean the Mavs are better team; it just means that they played better over that six-game span.
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 28, 2011 11:21 AM EDT reply actions 2 recs
The funny thing is
that in asserting that the Heat were statistically superior to the Mavs, he’s intrinsically assuming that basketball statistics function basically the same way as baseball statistics — and they don’t. As M. Haubs points out, there were some strong statistical indicators favoring the Mavs over the Heat in that series. (True, most people didn’t take the Mavs all that seriously, but as John Hollinger showed in some detail, that’s not because the indicators weren’t there.)
by The Ancient Mariner on Jun 28, 2011 2:29 PM EDT up reply actions 1 recs
Here's the deal --- and I question this every time when one of these goes up --
which sabermetrics sites are THEY reading?
Have any of you ever read that an optimized lineup will overcome the emotions of a petulant Boras rookie thinking he should be higher in the order? Have any of you EVER read that stealing or bunting is NEVER a good idea? Would any of us ever dare suggest that the established aging curve takes notable exceptions for the greatest players of all time? Can any of us agree on the effect of steroids vs. juiced balls?
So many of these type of articles are giant red herrings arguing against that which we’re not really disputing. We know there are assholes, and we know that personality management is something kinda important when dealing with multi-million-dollar type A jocks. Aaaaaand we know that there’s a lot we can’t measure.
Nobody is saying sabermetrics answers everything. Nobody.
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 28, 2011 11:35 AM EDT reply actions 6 recs
edit:
*Would any of us ever dare suggest that the established aging curve DOESN’T take notable exceptions for the greatest players of all time?
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 28, 2011 11:51 AM EDT up reply actions
Great point. I feel like anti-sabermetric pieces often stoop to this method and pose conclusions that sabermetricians do not claim to make. If someone makes the effort to understand these stats and what they measure and choose not to buy in, I am okay with that. But to just dismiss with a small understanding of what advanced stats try to measure is just unprofessional and careless.
Side note:
What’s with lumping the fantasy guy/sabermetrics guy together? There’s a lot of crossover, but damn if we’re not talking about two very different audiences.
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
Yup.
I didn’t realize that fantasy leagues used any statistics beyond the ones that you find in any box score. Obviously if you like baseball numbers, you’ll probably like fantasy baseball, but the thought processes used for real baseball and fantasy baseball are pretty different.
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 28, 2011 11:37 AM EDT up reply actions
The thing is,
it’s not very fun to use sabermetrics in fantasy, or the ones I’ve run all kinda got boring when all the rate statistics regressed to the mean around July.
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 28, 2011 11:39 AM EDT up reply actions
I've never really gotten huge into fantasy
I like the BOAB league, because it doesn’t require constant engagement and I’d rather evaluate my picks by WAR than HR/RBI/R/BA/SB, but I’ve always found baseball interesting without the added competitive feel of running a fantasy team.
Fantasy definitely adds that level of uncertainty because stats like RBI, R and BA are so dependent on things that won’t always level out over the course of a season, so you can just get lucky and feel like you made a great pick.
I don’t know why there aren’t more WAR fantasy leagues popping up, although I suppose that maybe I’m just not hip to that beat.
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 28, 2011 11:53 AM EDT up reply actions
Maybe it's just me, but I would like to point out that
I’m still fond of counting stats. Doubles, Ks, HRs, etc.
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 28, 2011 11:56 AM EDT up reply actions
Yup, I still sort through those numbers all the time
Just because I prefer ISO to grade power doesn’t mean that I don’t appreciate a nice HR total or whatever. Counting stats certainly have their uses.
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 28, 2011 11:57 AM EDT up reply actions
Speaking of which,
you notice the leader is Bautista at 23? “On pace” (or w/e) for 46. Feels small, doesn’t it?
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 28, 2011 11:59 AM EDT up reply actions
Feels small compared to what he was on pace for a month ago
before he only hit 4 HR in June
I LIKE IKE!
Fantasy
This really stuck out to me too.
Bullpen Banter's Top 100 Prospects for 2011!
- - - - - - - -
You can find my musings at Bullpen Banter and Beyond the Box Score.
Said this on the other post, but
The Mavs are apparently the most data-friendly team in the league. If this is really the only example he’s got, then he’s really not even standing on one leg.
by notoutofthe on Jun 28, 2011 11:58 AM EDT reply actions 1 recs
Yeah, the Mavs example felt like a really odd choice to make
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 28, 2011 12:06 PM EDT up reply actions
That is indeed serious ignorance re: the Mavs
and you don’t have to pay attention to basketball stats geeks to know about this, you just have to read FanGraphs.
by The Ancient Mariner on Jun 28, 2011 2:16 PM EDT up reply actions
I believe Lehrer followed up on this in a Wired column.
A few people have suggested that I erred in choosing the Mavs as my primary example because they relied heavily on statistics during the season. But that was my point! Here is the most “forward” thinking team in the league, and yet when it came to their lineup in the last three games of the Finals they chose to start a player who, at least according to every conventional statistical analysis, didn’t look so hot. But Carlisle wisely realized that Barea had other things going for him
Meanwhile Marc Cuban said:
"Roland [Beech of 82games, a “stathed”] was a key part to all this. I give a lot of credit to Coach Carlisle for putting Roland on the bench and interfacing with him, and making sure we understood exactly what was going on. Knowing what lineups work, what the issues were in terms of play calls and training."
There’s a good chance Beech was actually the one who recommended putting in Barea. Lehrer basically ignores the fact that the team he is using in support of his argument is actually more statistic reliant than any other team in the NBA.
"You can spend minutes, hours, days weeks or even months overanalyzing a situation; trying to put the pieces together, justifying what would’ve, could’ve happened – or you can just leave the pieces on the floor and move the **** on."
-Tupac Amaru Shakur
by NetsMets4Life on Jun 29, 2011 2:57 PM EDT up reply actions
I think the popularity of Moneyball partially created this false dichotomy
of “statheads” vs. traditional scouting – who’s right and who’s wrong. That’s not meant to be an indictment of the book. But, I think that’s the takeaway from a lot of folks who read it (and this seems to be reflected in the trailer for the film).
"It wasn’t that it was slippery or anything like that. It was just, dadgum, my hands and the balls were so wet..." - Tim Hudson
by KoKo the Monkey (T-Bone) on Jun 28, 2011 12:15 PM EDT reply actions 1 recs
in addition...
As Justin points out, there is some work/internet commenters that proliferates bad sabrmetrics that is devoid of critical thinking. I’d wager many writers/people couldn’t tell you the definition of the word or its purpose.
Bullpen Banter's Top 100 Prospects for 2011!
- - - - - - - -
You can find my musings at Bullpen Banter and Beyond the Box Score.
by JD Sussman on Jun 28, 2011 12:25 PM EDT up reply actions 1 recs
Dear Mr. Lehrer,
I’m really confused.
What are you railing against? A part of your article makes sense — the application of a sabermetrics in a basketball/football idiom — but the rest of it…well, just…yeah. Just keep going.
If you’re making the point that the same sort of analysis that works in baseball doesn’t necessarily work in other sports, fine. I totally agree with you there, dude. Each sport should be treated separately. You see, if you’ve noticed something about sabermetrics, it’s that we often talk about the lack of necessitated team-oriented play in baseball, which sets baseball apart from other team sports in this regard. No, no. I totally agree that we can’t measure everything. Nobody can measure everything. Physicists can’t measure everything that physically happens, and they’ve been going a lot longer than we have.
But there are so many other flaws with your thought process (Sabermetrics is like buying cars?
Really?
And you say you’re an editor? For Wired? The tech magazine?
Hey, I’m a programmer. Do you think we programmers and other IT professionals buy computers based entirely off the GHz number on the processor?
No? You think we know something special about the very things we work with on a day-to-day basis, and have had some particular training, perhaps an education that confers upon us the ability to make better informed decisions than some schlep walking in off the street?
Oh, I see. Because your article makes it sound like completely uninformed random people with no knowledge whatsoever are tantamount to those with years, possibly decades of hands-on experience in precisely the field you’re questioning.
Lord, this was a really long aside. Okay, we’re going back now.)
I do believe that the notion of defense-independent pitching (whether or not you agree with the underlying premise aside, it is a radical idea) came from sabermetrics and is completely counterintuitive to the way that most people experience the game of baseball.
What sort of evidence do you have that leads you to believe that similarly educated people in similar roles in other sports would just walk themselves off a mathematical and scientific cliff? The difference between your atrocious car analogy and the sabermetric analogy is that WE USE SCIENCE. (I know you’re reading this, but just imagine someone like R. Lee Ermey screaming this at you, because hey, it’s kind of important.)
We did not walk into a car dealership without formal training about what to measure and what to not measure and what is immeasurable about cars. We are not random people who walked off the street and were given some faux stock market data to study. We are people who have self-selected for this because we have questioning minds. We are a community of people who question our results.
Stop it.
Cordially,
Some jerk whose post you will never read.
by jwiscarson on Jun 28, 2011 12:33 PM EDT reply actions 3 recs
sub-atomic level physics
is heavily reliant on context and probabilities as well. Look up Schroedinger’s cat. Also according to physics, essentially anything is possible including walking through a brick wall, so even though the odds of that happening are something ridiculous like 1 in 10^235, if you tried once a second for the entire history and future of the universe you might be able to walk through that brick wall, and today may be your day.
To the true point: nothing is infallible, not even Einstein theories. But sabermetrics do a very good job of approximating player talent levels and luck.
brick wall thing is from the Elegant Universe by Brian Greene, a crazy good book on string theory.
by roons11 on Jun 29, 2011 8:49 AM EDT reply actions 3 recs
I want you to post more around here.
Managing Editor at Beyond the Box Score and MLB Daily Dish. Follow me @justinbopp
by Justin Bopp on Jun 29, 2011 10:21 AM EDT up reply actions
Yeah, this comment really made me think
Yes, my real name is actually Satchel.
I'm a columnist for Beyond the Box Score and a writer for MLB Daily Dish.
Oh, I'm on Twitter, too.
by Satchel Price on Jun 29, 2011 8:53 PM EDT via mobile up reply actions
fun fact
Brian Greene has an Erdos number of 3 and a Bacon number of 2. (Unlike some contenders, his Bacon number is totally legit, as it comes a movie in which he had a speaking part and was not playing himself).
Not actually affiliated with whygavs.
by WHYG Zane Smith on Jun 29, 2011 9:05 PM EDT up reply actions
The Mavericks, meanwhile, heavily relied on statistics on order to improve their team.
And now they are champions. Lehrer even mentions this. He basically takes a dump on his own argument. Repeatedly.
"You can spend minutes, hours, days weeks or even months overanalyzing a situation; trying to put the pieces together, justifying what would’ve, could’ve happened – or you can just leave the pieces on the floor and move the **** on."
-Tupac Amaru Shakur

by 




























