If you are a regular reader of BtB then you will, hopefully, have read my article last week which showed why we shouldn't count on any modern day player breaching the artificial, yet magical .400 barrier. If, like me, you have an amoebic memory then it's worth remembering that since 1900 there have only been thirteen .400 seasons, which have been compiled by a meager 8 players. More interestingly the last time this feat was achieved was in 1941, just as World War II was kicking off. What is going on? Why has the .400 season entered the pantheon of distant memory?
There are a bunch of cool graphs in here which Alex Reisner has kindly let me nab from his site. So a big thank-you to Alex. Please check out his wares.
One popular point of view is a theory expounded and popularized by Stephen Jay Gould (in his book The Spread of Excellence), who argued that over time the amount of variation in a population falls. Be it the tallest and shortest members of a species or the best and worst students at a school, the difference between the extremes will gradually narrow. If we apply this theory to batting average we would, over time, expect evolution to dictate that the worst hitters would either improve or be expunged thereby creating a clump around the mean. Has this happened? Fortunately it is easy enough to test. We can simply work out the standard deviation of AVG for each year and see how it has changed.
Standard Deviation of AVG
The trend is clear: standard deviation has been falling continually. At a first glance this would appear to confirm Gould's hypothesis that the skill levels of batters are converging. So while we don't see any .400 hitters neither do we see as many .200. The problem with looking at standard deviation is that it only measures the amount of spread in a population. No more, no less. All we can conclude from a falling standard deviation is that the spread in the data is reducing. We can't necessarily pinpoint why.
Saying that, it hasn't stopped analysts from putting forward countless reasons as to why variation has fallen. This ranges from the obvious such as, more competition, better pitching, improved diet and conditioning, to the obscure, like improved park consistency and more night games (yes it is true). Trying to decipher which of these is real is hard. Before we get lost in the intricacies of debate consider the basic tenet of baseball: the batter pitcher confrontation. For standard deviation to have narrowed one of three things must have happened. Either poor batters have been weeded out, or pitchers have become more consistent, or both. Factors such as diet, conditioning and whatnot should contribute equally across pitching and hitting. Just because there are fewer players chugging down a couple of Big Macs between games it doesn't mean to say that they have improved as ball players. In fact, baseball is one of those rare sports where diet and conditioning are probably relatively less important.
One intriguing thought is that if Gould's hypothesis is right and hitters improve why don't more players breach the .400 mark? Well, the obvious answer is that pitching must have improved too. The question is can we decipher which factor has been more important? At a basic level this can be measured by plotting mean AVG over time. To take an extreme, if pitching stood still but all batters improved we'd expect AVG to increase year on year. If, on the other hand, the 1920s and 1930s was an era of superlative batting, then pitching could remain at the same skill level and we could still account for a drop-off in AVG.
Mean of AVG
The graph shows that after peaking in the 1920s and 1930s, which incidentally is in the middle of the era of the last .400 hitter, AVG took a dive through the post war period until staging some sort of recovery in the mid-1990s. Let's focus on the trough and see if we can glean anything useful. Clearly the greater the distance between the mean and a target, the harder it is to achieve the target; so the decline in AVG has meant that achieving a .400 mark is incrementally harder. Indeed we showed this not so startling conclusion in the previous article. But what has caused this trough? Has batting skill really deteriorated? No serious analyst would give much credence to that particular theory; why would batters have got worse? Analysts and fans have a tendency to ignore contemporary greatness at the behest of past stars of the game. Indeed, to my mind there is no incontrovertible analytical way to answer the eternal question as to how players from different generations compare (though many have tried). But consider how the nature of the power game has changed since Ruthian days. In the very early part of the 20th century making contact was considered the most important batting skill in baseball. As we know Ruth altered the axioms of the game. The home run became an everyday occurrence, and what is good for SLG isn't necessarily great for AVG.
Mean of SLG
Examining the graph of SLG over time we can see that it is much more stable than AVG insomuch as variation hasn't really fallen at all. At first this may be a tad surprising as it is both contrary to the Gouldian view and counter our contact-power assertions. Shouldn't we expect SLG to increase as AVG falls as the long ball becomes more common? Remember that SLG does have a contact component embedded in it. If we look at ISO we do indeed see that the power game has proliferated.
Mean of ISO
This is all well and good and is very much from a batter's perspective. But a homer for the batter means a ding against the pitcher. Has pitching improved at the same rate as hitting, thereby accounting for static SLG? Or has pitching improved incrementally ahead of batting resulting in depressed AVG? Perhaps the rawest measure of pitching performance is the K. The number of whiffs has shown an alpinean increase which would seem to indicate that either pitching has improved immeasurably or that in the ever increasing search for power batters are striking out more.
Mean of K
Plotting HR rate over the same period shows some visual correlation with K rate but not enough to suggest that it is the sole driver of Ks.
Mean of HR
It would seem that by upping the gas that hurlers have shown a tendency to give up more long balls (although the most likely explanation it is worth bearing in mind that it is still conjecture as correlation does not necessarily imply causation). As pitching has changed irrevocably since the end of the dead ball era there are also a bunch of other factors that need to be considered. In the modern game it is acceptable to pitch around stronger hitters, use relievers more frequently so batters don't see the same pitcher all game, and take advantage of the platoon - all of these were anathema in bygone days. Whichever explanation we promulgate we arrive at the same conclusion: both pitching and hitting have significantly evolved, which has conspired to hold down AVG at the expense of other metrics. This is best shown by considering the variation in power stats such as SLG for batters and K/9 for pitchers.
Std dev of ISO
Std dev of K
Back to the original question: Is Gould right? The weight of evidence forces us to err on the side of doubt. It is actually very difficult, if not impossible, to decouple changes in skill to changes to how the game of baseball is fundamentally played. Although variation in AVG has certainly fallen it would appear that this has been substituted by the rise of the power game. Indeed, variance here, if anything, has been steadily on the increase. And why should we focus on AVG as a harbinger of diminishing skill differentials? If we looked solely at ISO we would conclude that Gould is wrong.
Also if we look at other sports it is difficult for us to support Gould's theory. Take cricket, where we haven't observed the coalescence of batting averages that we have witnessed in baseball. (Cricket, for those of you that don't know, hasn't seen the revolution in bowling that baseball has seen in pitching.) To my mind changes to variation appear more closely linked to fundamental changes in how the game is played and not necessarily collective improvements in the population.