Evaluating relievers for Hall of Fame consideration

I've written several posts discussing the 2015 Hall of Fame ballot, and one of the things that struck me was the extreme difficulty in evaluating relievers. Several factors require serious consideration:

• Treatment of role players -- A typical closer pitches 60-70 innings a year out of around 1,500. Do they deserve the same status as starters who pitch far more innings or the position player who plays daily? With only four relievers in the Hall so far, it appears to be an open question.
• Separating players -- Mariano Rivera set the standard for relievers, not just for his generation but for all of baseball history. Subsequent voters will have to work hard to not say to themselves "Well, Pitcher X, he was no Mariano Rivera." That's not a fair standard -- it's like saying Ken Griffey Jr. was no Willie Mays.
• Proper comparisons -- My pet peeve has always been comparisons between apples and oranges, so in my mind, the proper comparison is between relievers and relievers, to see how they compare to each other and to not lump them with other pitchers or position players.
• Use of the closer -- In today's game, the closer only appears in save situations, absent extenuating circumstances. This means his team is ahead and he enters the game with the bases empty. I'm not suggesting this is easy, since there's plenty of pitchers who have shown just how difficult this is, but does the number of saves denote achievement, let alone greatness?

This is from a Tableau data viz (view it yourself here):

The Bill James Hall of Fame Monitor and Career Standards depict how players should be rated, but they don't work for relievers since James never created criteria specific to their role. Earlier this year, Beyond the Box Score Associate Editor Ryan Morrison wrote a post that proposed using RE24 as a measure of reliever effectiveness. This is the definition from the FanGraphs glossary page:

RE24 (runs above average by the 24 base/out states): RE24 is the difference in run expectancy (RE) between the start of the play and the end of the play. That difference is then credited/debited to the batter and the pitcher. Over the course of the season, each players’ RE24 for individual plays is added up to get his season total RE24.

For closers, every at-bat is important, since in modern usage they're only in the game if it's a high leverage save situation. If RE24 can work for a given year, it can certainly be used for career evaluation purposes as well. This table shows the top relievers in RE24 since around 1940:

rIP = relief innings pitched BF = batters faced BS=blown saves SD = shutdowns MD= meltdowns.More data can be seen in this Google Docs spreadsheet

Games shown are relief appearances only, so for example, Mariano Rivera's data doesn't include the ten games he started at the beginning of his career. There are some surprises (Keith Foulke?), but generally it helps show the distinction between effective relief and just saves, since even in the brief history of the save, closer utilization has changed.

This will be the 13th year Lee Smith is on the Hall of Fame ballot, and this is the HOF voting pattern:

He had been building momentum, but ran into a brick wall in 2014. Viewing him using RE24 places him right up there with every other top reliever in baseball history not named Mariano Rivera, and look at the other players around him -- those players will give HOF voters fits when their time for consideration comes. I tend against enshrinement for Smith, since his was more a lengthy career than one marked with a sustained run of being considered among the best closers in the game.

We need better ways to evaluate relievers than saves. RE24, as Ryan suggested in his post, shouldn't be used by itself (no stat should), but it does an excellent job of cutting across time and showing who the effective relievers were.

All data from Baseball-Reference and FanGraphs

Scott Lindholm lives in Davenport, IA. Follow him on Twitter @ScottLindholm.