Let me say this up front: I like Jonah Lehrer. I am a fan of his writing and generally find him quite informative and thoughtful. This probably explains my visceral reaction to his latest, a dismissive column at Grantland that criticizes the use of analytics in various sports, including baseball.
As JD mentioned, it's a bad column and disappointing. I don't simply claim it to be bad because I disagree with his conclusions. I say it's bad because it is sloppy and Lehrer is a smarter guy than this.
The argument in short:
My worry is that sports teams are starting to suffer from a version of the horsepower mistake. Like a confused car shopper, they are seeking out the safety of math, trying to make extremely complicated personnel decisions by fixating on statistics. Instead of accepting the inherent mystery of athletic talent — or at least taking those intangibles into account — they are pretending that the numbers explain everything. And so we end up with teams that are like the worst kind of car. They look good on paper — so much horsepower! — but they fail to satisfy. The dashboard is ugly, the frame squeaks, and the front seats make our ass hurt.
First, there isn't a single team nor analyst that claims that sabermetrics capture all there is to know about athletic talent and performance. Teams use a variety of other metrics and method to assess player talent as well as likelihood to succeed.
Second, and more important, Lehrer's main argument shouldn't be that teams are assembling bad teams because of a narrow-minded focus on things they can quantify. The argument should be that teams that don't think deeply about what are the right metrics and how much variance they account for in player achievement will fail just as much as those teams that used to generally ignore analytical approaches to the game.
Data and statistics are not to blame for bad decisions--their misapplication is. Lehrer is trying to get at this point, but his misinformed broadside against sports analytics makes it real easy to miss.
Moreover, Lehrer's examples are just plain bad.
One example he uses is the NBA Champion Dallas Mavericks. He claims that because the Mavs did not match up statistically against each of their playoff opponents, yet prevailed, shows that a reliance on these statistics is flawed.
Lehrer is both right and wrong.
He's right in that relying solely on these statistics and not taking into account context would be a mistake. Again, I am waiting for someone to raise their hand and say they advocate this, particularly on a game by game or series by series basis.
He's wrong in that this example in no way proves or disproves the usefulness or accuracy of the underlying analytics. The playoffs are a handful of 7-game series. Anyone with a basic understanding of probability knows this is the last place you would look to prove or disprove the effectiveness of such measures given the role of randomness and luck.
Finally, Lehrer continually claims that a long list of intangibles largely determines the outcomes of games. He goes so far as to say that:
not everything that matters can be measured, and that success in sports (not to mention car shopping) is shaped by a long list of intangibles.
Now, he may very well be right, particularly if we are trying to make point predictions. But my first question is, if it can't be measured how do we know that it affects outcomes they way he claims? It's like someone who sees a UFO jumping to the conclusion that since we can't explain what it is it must be aliens. If it's unidentified you have no justification for making that leap. In Lehrer's case, neither does he.
Furthermore, sabermetrics--and any analytics whose subject is people and not atoms--is all about context and probabilities. When players don't perform as expected it could very well be for "intangible" reasons. This doesn't negate sabermetrics, not one bit. It goes hand in hand.
Like with any technology, sabermetrics and other analytical approaches won't by themselves lead to optimal decision making. Their is a mindset that must accompany them. That's a message worth delivering. Unfortunately, Lehrer is the wrong messenger. At least, he was on this day.