Anatomy of a Luck vs. Skill Evaluation

When reading saber-slanted writing you will often find writers debating whether a player's accomplishments in a given season are the result of skill or luck.

The discussion goes something like this: "Player X is having a fantastic year at the plate, but his offensive statistics are largely the result of luck since his (insert favorite peripheral stat of choice) is well above average." Batting average on balls in play is arguably the most common statistic used in this fashion.

The discussion can be applied to pitchers as well, and the pronouncement of luck can also take many forms (i.e. the player is lucky/skilled since their BABIP is higher/lower than the average).

But in truth it is more complicated than that. It's one thing to go down a level of analysis beyond the mere descriptive statistics of OPS or wOBA, quite another to drill down further to ensure that the shortcut one uses to pronounce a player lucky actually holds in a given situation.

We are lucky enough to have discovered a number of generalizable findings in the field of baseball that provide great context when trying to evaluate player performance. The problem is that sometimes we get lazy and deploy these findings too quickly when analyzing whether a player's performance is reflective of their true talent or simply the result of randomness.

To flesh out this idea I would like to present an anatomy of how one might make a luck versus skill call. I'll use Tampa Bay's Casey Kotchman and his .367 wOBA as an example.

The first logical place to start with Kotchman is with his batting average on balls in play, or BABIP. We know from some great research that, on average, 30% of all balls put in play wind up going for hits. Amazingly enough, this relationship holds even when we look at position players as pitchers.

This season, Kotchman sports a BABIP of .356. If we take his production this year and simply regress it to a .300 BABIP he drops from a .367 wOBA to roughly .328.

So the easy thing to do is say that, based on Kotchman's high BABIP, the Rays' first baseman is having a great year, but it's fueled by luck. At some point, even if that isn't until next year, balls will not bounce Kotchman's way and he'll return to being an average hitter.

But simply having a high BABIP doesn't mean performance is fueled by luck. We have to go further.

The first stop down the analytical ladder is Kotchman's historical performance, especially his recent history. Kotchman has a career BABIP of .281, so he is not one of those hitters with consistently higher BABIP's than the general .300. Coming into this year, he never managed a BABIP over .305. 

To give you a sense of how much BABIP can skew batting results we need look no further than Kotchman's 2010 campaign. Last season, Kotchman managed a feeble .270 wOBA based on a BABIP of .229. However. If we simply gave Kotchman a .300 BABIP instead his wOBA vaults to roughly .334. Much more respectable. Given that, Kotchman was identified as a potential bounce-back candidate heading into this season. 

So since history doesn't appear to be on Kotchman's side, maybe this whole thing is largely fueled by luck?

But maybe something has changed with Kotchman. Maybe he's made some changes in his mechanics and approach that account for the higher BABIP.

The next stop down the analytical ladder would be batted ball distribution. We know that the average BABIP of .300 is the result of different batted ball types (i.e. Ground balls, fly balls, and line drives) falling for hits at different rates. Generally, 24% of ground balls, 15% of fly balls, and 73% of line drives fall for hits. Now, imagine a batter drastically increases their line drive rate. Even a modest increase would boot their BABIP and, therefore, there overall production.

Last year, Kotchman's .229 BABIP was the result of 17.5% LDs, 55.4% GBs, and 27.1% FBs. For his career, Kotchman's BABIP per batted ball type looks like this: .715 LDs, .193 GBs, and .170 FBs. Based on those numbers, a quick back of the envelope calculation had Kotchman's expected BABIP at .278. This year, Kotchman has increased his LD% by roughly 2%. However, the expected increased in BABIP is only .010 (.288 vs. .278) based on those numbers. Additionally, Kotchman has generally underperformed his expected BABIP by an average of .011. So the fact that he is outperforming his xBABIP this year by .067 should raise suspicion.

But maybe the fact that he is outperforming his xBABIP and his career averages is the result of some environmental change, like switching ball parks. Maybe the talent was there all along, but playing in parks that depressed offensive production simply hid Kotchman's ability.

From 2004-2008 Kotchman played for the Angels. Between 2008 and 2009 Kotchman played for the Angels, Braves, and Red Sox. Last year, Kotchman played for the Mariners before signing with the Rays in 2011.

Unfortunately, the change in parks does not seem to explain Kotchman's BABIP. The average difference in hits per ball in play across these parks does not equate to the difference between Kotchman's expected and actual BABIPs. It's true that pitcher-friendly parks have generally depressed his BABIP and playing a full season in Tampa Bay generally will boost BABIP, but it's hard to imagine it is accounting for the 14% over achievement.

Finally, maybe Kotchman has made some improvements in his platoon splits or is simply hitting against more right-handed pitchers.

For his career, Kotchman sports a BABIP of .277 and an OPS of .757 against RHP and .295 and .672 against LHP. He has faced RHPs 77% of the time during his career. This year, Kotchman still has the reversed BABIP split, with totals of .352 and .873 against RHPs and .365 and .747 against LHPs. He's faced RHPs 75% of the time this year. So while Kotchman is hitting RHPs about 15% better than his career average, he's actually faced them 2% less of the time. 

Weighing all the evidence it would appear that, yes, a significant amount of Kotchman's performance this year can be attributed to beneficial randomness given that his BABIP isn't explainable as either latent talent or the result of specific changes to his talent. He's certainly a better hitter than 2010, and moving to Tampa will boost his production a bit given how the park plays, but Kotchman is likely to regress back to a league-average hitter going forward.

Now, of course, we could go even deeper. Taking a look at film and breaking down Kotchman's mechanics might reveal a change that the numbers haven't. It's also possible he's taking a different approach, a la Jose Bautista, and either pulling the ball more or going the other way. (A quick check of the numbers suggests this isn't the case, as hit percentages to field are basically in line with career averages.) These are just a few examples. There are other ways to conduct this analysis and, arguably, different conclusions one might draw.

The point of this (extremely long) exercise is that the rules of thumb we have uncovered and developed over time as baseball analysts often hold true, but before accounting for over- or -under achievement with luck we need to go further down the analytical ladder to be as confident as possible, especially when dealing with individual players.

-----------------

Data: FanGraphs and Baseball-Reference

BABIP regressions calculated using the In-Season Batter Regression Tool