Consistent bullpens and unicorns

Bullpens are notoriously difficult to evaluate. If you've spent any time at all poking around saber-oriented baseball sites, this is probably something you're aware of. Relievers are volatile on the whole, partly due to the nature in which they appear in games and partly due to the fact that they tend to be less talented pitchers overall than their starting counterparts. Because of these factors, among others, they can be tough to count on.

So I decided to do a little poking around and determine which bullpens have been the most consistent from month-to-month over 2014. While shorter time intervals might be helpful, a monthly breakdown should suffice for the time being. To evaluate team bullpen performance, I used FIP- over each month, and tracked the change in bullpen FIP- as the season progressed.

Remember, FIP only accounts for thing that pitchers control: walks, hit batsmen, strikeouts and home runs. Relievers tend to strike out batters at a higher rate than starters, but also walk more batters. They benefit from the platoon advantage at a higher rate than starters, but due to the lack of innings, can be significantly dinged for allowing dingers. FIP- captures these things and nothing else.

Below is a table of reliever performance for each month of the season. Keep in mind, each point of FIP-  above 100 (i.e.. 101, 102, etc.) represents one percent worse than average while each point of FIP- below 100 (i.e. 99, 98, etc.) represents a percent better than average. In essence, the lower the number, the better.

By a simple average, it's easy to see who the "best" bullpens have been by FIP-. But that's not exactly the question I've posed here. Instead, I wanted to know which bullpens have been the most consistent. The chart below shows the change in performance of each bullpen during the first four months of the season. Have a look:

Click to expand

So, it's important to note the meaning of the word "consistent", and mention that "consistent" doesn't necessarily mean "good." It just notes that the variance from month to month follows a predictable pattern. Given that we only have five months to analyze, that pattern is surely limited, but a pattern can emerge, and some do. However, it doesn't mean that the pattern observed over this time is likely to continue. Instead, it's simply an observance. Let's identify some patterns.

Consistently Good

Washington Nationals -- The Nats show some of the lowest average variance from month-to-month and have a FIP- that's tops in the game. Despite the struggles of Rafael Soriano, guys like Tyler Clippard and Craig Stammen have buoyed a very effective bullpen.

Texas Rangers -- The Rangers bullpen has performed reliably average or better despite losing Joakim Soria and getting inconsistent work out of Neal Cotts. Texas stinks this year, but the bullpen remains functional at worst.

Seattle Mariners -- This is why using five months of data can be scary: The Mariners have consistently performed way above average. No one doubts that it's a talented ‘pen, but these look like some unsustainable numbers no matter who's pitching. At least they've been steady about it ever since a rough April.

Consistently Bad

Detroit Tigers -- This isn't good for any contender. The Tigers' best two relievers in the first half, Joba Chamberlain and Al Alburqureque, have been their two worst in the second half. Add Joe Nathan's struggles and this bullpen has problematic all year long.

New York Mets -- The bullpen in Flushing has been consistently worse than average, although we've seen some swings in just how bad it has actually been (April! August!). There just aren't a ton of talented guys in the fold at the moment for the Mets.

Consistently Inconsistent

Minnesota Twins -- Casey Fien and Glen Perkins have been dependable, but after that it's a hodgepodge of ups and downs. There haven't been a lot of leads to protect in Minnesota, but the bullpen surely hasn't helped the team's cause at times.

Milwaukee Brewers -- Perhaps no bullpen has been more two-faced than the Brewers'. The ups and downs from Francisco Rodriguez and company are just killing the Brewers' chances of the bullpen (although they aren't the only unit struggling).

St. Louis Cardinals -- The Cards have had shaky underlying performances drive their shaky bullpen. Kevin Siegrist, Carlos Martinez and, to a lesser extent, Trevor Rosenthal are responsible or making late innings more eventful than they should be without a moment's notice.

Pittsburgh Pirates -- Pittsburgh's relief struggles have been well-documented. Mark Melancon, Justin Wilson and Tony Watson are all talented, but Ernesto Frieri just got himself DFA'd and Jason Grilli had been unreliable before him. The rest of the bullpen staff is seemingly replacement level.

San Diego Padres -- In San Diego, things look pretty good on the surface: a balanced monthly variance and a strong FIP- over the course of the year. But looking at the graph above, we can see some swings in monthly performance that, while they balance themselves out when averaged, showcase inconsistency.

As we can see from the graph and the notables above, there have been some consistent bullpens. Some are good, some are bad, and others are consistently all over the place. But these figures showcase bullpens as a whole, and after diving into one particular bullpen, we see that the underlying results of a "consistent" bullpen can truly be anything but consistent.

The Arizona Diamondbacks have been one of the more consistent bullpens from month-to-month, but digging into individual performances, they've been quite toxic at times. As I noted in detail at Inside the ‘Zona last week, D-backs relievers have been all over the place.

Although Arizona's had a relatively consistent bullpen, we've seen some big changes as the year's worn on. Closer Addison Reed has been wildly inconsistent, as you'd expect having watched his performance this year. Brad Ziegler‘s even had his struggles, and uncharacteristically so. Oliver Perez has been the most consistent reliever and Evan Marshall has gotten better as the season's progressed. Randall Delgado has been consistently around average while Matt Stites has had his struggles and Eury de la Rosa had a poor July and a good August. When we look at the parts rather than the sum, the "consistent" bullpen has certainly had its ups and down.

I won't reproduce the graphs from the original posts here, but know that each reliever has had significant bumps in the road this season. And I would surmise, without digging into all 30 bullpens, that this is pretty common. Ask any manager what they'd most like out of their bullpen and I'd guess that "consistent performance" ranks near the top of the list. For example, Addison Reed's FIP- on the season is 86. But he's spent three months well below average only to salvage it with a filthy August. I don't think this is at all groundbreaking information, but rather confirms what we know: Relievers are volatile, and Reed is the quintessential example of this.

So how do we actually evaluate consistency of a bullpen? Taking a team-wide angle seems too simplistic and captures way too much noise. A reliever-by-reliever approach might make the most sense, but is pretty granular, which might be fine although it may make it hard to explain and/or relate to those without a love for plowing through relief numbers. Something in the middle seems necessary, although I don't have that answer at the time. It's just that this brief study has reinforced something we've known for a long time. When it comes to bullpens, they aren't always what they seem, and just when you think you've found a "consistent" bullpen, you may have found anything but.

. . .

Jeff Wiser is a featured writer at Beyond the Box Score and co-author of Inside the 'Zona, an analytical look at the Arizona Diamondbacks. You can find his work on craft beer at BeerGraphs and follow him on Twitter @OutfieldGrass24.