Introducing ARJS

by Dan Turkenkopf on Jan 2, 2010 8:24 AM EST in Positional Reviews

The other day on Twitter, Sky, and I were discussing how to value relievers. The conversation was triggered by a comment from Colin Wyers where he said an average reliever would successfully complete 82% of all save situations a closer finds himself in. Sky then used Mariano Rivera's save percentage since 2007 to do a quick and dirty WAA calculation.

Somehow that led into how we'd measure reliever success using Win Probability while still keeping in mind bullpen chaining. I tossed out the idea of WPA / Leverage Index for each plate appearance times the chained leverage index value for when the pitcher entered the game. Sky suggested that we should readjust the chained leverage index for each inning.

All this rehash of our Tweeting leads up to my introduction of ARJS - Another Reliever Junk Stat. I don't know if it has any meaning beyond WAR or WPA but I told Sky I'd run the numbers.

I'm looking for you guys to tell me whether this is stupid or if you think it has merit. Results and the formula after the jump.

Here's the top 10 and bottom 10 in ARJS for 2009:

Jonathan Broxton	4.96
David Aardsma	4.57
Andrew Bailey	4.22
Huston Street	4.15
Joe Nathan	4.01
Mike Wuertz	3.78
Mariano Rivera	3.74
Trevor Hoffman	3.65
Heath Bell	3.61
Jonathan Papelbon	3.61

Brad Lidge	-1.33
Rafael Perez	-1.03
Jesus Colome	-0.89
Chris Ray	-0.89
Brian Wolfe	-0.88
Scott Linebrink	-0.85
Logan Kensing	-0.71
Felipe Paulino	-0.66
Luis Ayala	-0.62
John Bale	-0.53

The method to calculate ARJS is:

Determine all relief appearances.
For each relief appearance, calculate the leverage index at the time of entry by inning. That means if a reliever comes in with the bases loaded and two outs in the eighth, and stays in for the ninth, we'll have two data points; the bases loaded, two outs in the eighth and the nobody on, nobody out in the ninth.
Account for bullpen chaining using the formula (1 + LI) / 2 = chained LI
Calculate WPA / Actual LI (WPA/LI) for each of the plate appearances against the pitcher.
Multiply the WPA/LI value by the chained LI value.
Sum those values across the entire season.

If anyone's interested in more details or the full result set, let me know in the comments and I'll try to oblige.

Tweet Comment 26 comments | 0 recs |

Do you like this story?

Comments

How do these numbers compare with straight WPA?

Which relievers show the biggest differences?

Also, why not re-adjust the chained LI for every plate appearance? If a reliever comes in with no outs and a runner on first, then gets a double play, he’ll get big WPA credit, but he probably doesn’t deserve the initial LI of the inning when he records the third out.

= =

I’ll add that part of this discussion was the idea that Mariano has value beyond his save percentage because he enters tie games often (which aren’t save opportunities) and faces tougher save opportunities than the typical reliever. Whether or not those claims are true, I’m always in favor of the more continuous metric (WPA) than the black and white one (saves).

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jan 2, 2010 8:40 AM EST reply actions

Also...

Is there any mathematical difference between doing WPA / pLI * (1+pLI)/2 on a seasonal level and doing it for each individual event and adding them all up?

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jan 2, 2010 4:04 PM EST up reply actions

There'd be no difference if the pitcher had their average performance every time out

But because sometimes the pitcher will do well in extremely high leverage situations and poorly in low leverage ones (for example), calculating by plate appearance can look substantially different.

by Dan Turkenkopf on Jan 3, 2010 7:19 AM EST up reply actions

WPA top 10:

Papelbon 5.13
Rivera 3.99
Nathan 3.89
Soria 3.57
Street 3.38
Thornton 3.12
Wuertz 3.05
Mijares 2.97
Affeldt 2.95
Broxton 2.94

by Alex Krolewski on Jan 2, 2010 4:04 PM EST up reply actions

WAR top 10:

Broxton 2.9
Thornton 2.5
Wuertz 2.4
Brian Wilson 2.4
Bailey 2.4
Hughes 2.2
Soriano 2.0
Rivera 2.0
CJ Wilson 2.0
Bell 2.0

by Alex Krolewski on Jan 2, 2010 4:06 PM EST up reply actions

I didn't want to readjust the chained leverage for each PA because then I'm double crediting the pitcher for his own success/ failure

If he gets himself into a jam he gets an initial hit on the WPA side, but then he gets the benefit of increased leverage if he gets out of it. My guess is that would make it look like a good outing when instead it was a poor one.

Using the chained LI at the time of entry gives the pitcher credit for how the manager chose to use him (I suppose his “reputation”) which is what I wanted to measure.

by Dan Turkenkopf on Jan 3, 2010 7:23 AM EST up reply actions

But a pitcher's total WPA for an appearance (well for each inning of an appearance)...

…will be the same no matter how he gets from point A to point B. Load the bases and strike three guys out is the same total WPA as striking three guys out from the get go. The first scenario will contribute to a higher average LI, though, but does that matter if you’re judging the pitcher event by event?

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jan 3, 2010 8:37 AM EST up reply actions

I think it does

Let’s use your scenario and assume that it’s the bottom of the ninth with the home team down by 1 run.

The home team’s win probability starting the inning is .2187. So that’s what both pitchers will end up with in WPA.

The pitcher who sets them down 1-2-3 has an average LI of 2.78 and each play has a WPA/LI of .026. In that case, the total WPA/LI calculated by event is close enough to summing WPA and then dividing by average LI – both roughly .078.

In the case of the pitcher who loads the bases first, his average LI is 6.625. If you calculate WPA/LI by plate appearance and sum them, you get -.012. If you take the total WPA and divide by average LI at the end, you get .033. That swings the appearance from a net negative to a net positive.

Looking at multiplying by the chained LI each event versus the entry point:

in the good case, Chained WPA = .149 and ARJS = .181
in the bad case, Chained WPA = -.103 and ARJS = -.028

Which is better? I’m not sure. It seems a bit harsh to lose a tenth of a win because you loaded the bases and got out of it, but maybe not.

by Dan Turkenkopf on Jan 3, 2010 11:24 AM EST up reply actions

I've thought about something like this before.

VEP brought up reliever measurements before and I’d come up with an idea similar to this in the past.

I’d add that a measurement like Rally’s defensive adjustment for pitchers would be good, so that we can get a measure of defense independence. That or pitcher zone runs for this would be nice. We’d have to switch those runs to wins of course.

I’m also interested in why you went with just chaining just entering LI instead of each PA LI. There’s something to that (the reason we chain reliever LI is because managers decide to chain relievers, so that methods only affects decisions by managers).

Marlin Maniac, a Florida Marlins blog
Come attend Intro to Sabermetrics 101!
Check me out at Beyond the Box Score as well.

by SFiercex4 on Jan 2, 2010 9:31 AM EST reply actions

Making ARJS defense-independent is going to be very difficult.

You could use PZR, but:
1. Only MGL has the play-by-play UZR data that you would need
2. More importantly, you run into some real sample size/measuring error issues with PZR for relievers. A SP has about the same number of defensive opportunities behind him as a full time player, but a reliever has only about a third as many defensive opportunities (and probably less, since relievers have a higher strikeout rate than starters).

Maybe you could do some kind of defense adjustment at the end of the process, regressing the reliever’s BABIP towards his career number and subtracting runs accordingly.

by Alex Krolewski on Jan 2, 2010 4:13 PM EST up reply actions

I think, given the win totals that we already have, Rally's method would be easier and perhaps a bit more reflective.

There’s also the issue of converting the runs to wins and keeping them in the same scale. I’m going to run some numbers on the Top/Bottom 10 using prorated UZR.

Marlin Maniac, a Florida Marlins blog
Come attend Intro to Sabermetrics 101!
Check me out at Beyond the Box Score as well.

by SFiercex4 on Jan 4, 2010 11:58 AM EST up reply actions

Should clarify.

That’s not an average reliever, that’s a reliever with average rate stats. That’s close to a replacement-level reliever.

by cwyers on Jan 2, 2010 10:39 AM EST reply actions

You mean league average pitcher, including starters?

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jan 2, 2010 11:08 AM EST up reply actions

Right.

by cwyers on Jan 2, 2010 11:24 AM EST up reply actions

relievers suffer from the problem that their usage is often not optimal, so that their quantitative stats are often dependent on incidental factors and thus do not reflect their value to a team accurately.

let’s say you have a 1.50 fip reliever who is not used optimally, ie only in the 9th inning. and now he is on the market. what is your team’s valuation of him? based on the work he’s done, maybe he’s not the best reliever in the league by value, but certainly a smarter team would get more out of him. having a shutdown reliever has value just by being there, because you have the option of using him. this is especially true in the playoffs as teams are more willing to use their ace relievers in critical situations and thus enhancing their value. some sort of usage independent stat for relievers should be useful especially in terms of team building.

by awayish on Jan 2, 2010 11:36 AM EST reply actions

Totally agree.

Teams can plug in relievers to their current specific context to determine the value.

But if you want to ignore team context, I prefer a “deserved LI” approach, outlined here. Good discussion in the comments, too:

http://www.beyondtheboxscore.com/2009/4/20/843972/what-leverage-index-do-relievers

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jan 2, 2010 12:23 PM EST up reply actions

that’s certainly a thought in the right direction. a variation of “deserved leverage” might be “estimated leverage by optimal use.” i would venture to say that for each team, the spread of late-inning leverage should be fairly consistent as a matter of statistical distribution. given the appropriate workload of a relief ace, we can calculate a leverage threshold for his usage. let’s say the reliever can appear in 65 games per year, and it is generally known that each year there are 70 late innings with leverage > 2.5. Then it would be reasonable to bring in the ace reliever once the leverage reaches 2.5.
doing things this way would present reliever value in a more practical and fair context.

by awayish on Jan 4, 2010 5:38 PM EST up reply actions

Brings up a good point. Smart teams pay for production and leverage it as they see fit. Dumb teams pay for leveraged production.

http://www.capitolavenueclub.com/

by PWHjort on Jan 2, 2010 1:11 PM EST up reply actions 1 recs

I like it.

Since ARJS accounts for chaining, it’s a better method to determine value above replacement than WPA; since it accounts for leveraged performance on a play-by-play basis, it may be superior to WAR.
Also, the ARJS top 10 list seems to have higher-quality relievers on it than the WPA or WAR top 10 lists.

by Alex Krolewski on Jan 2, 2010 4:19 PM EST reply actions

So there were two things I wanted to add on to ARJS

1. Defense independence (or some measure of it).
2. Replacement level instead of average.

I love the design Dan uses, and I think it may be on the way to figuring out what’s going on with relievers.

Here’s how I set out to do both things.

1. Rally’s method using team defensive metrics and prorating them for balls in play. I don’t think this will make a huge difference, but it’s worth looking at.
2. Add .03 wins/9 innings, assuming a 47% W% pitcher as replacement level for a reliever. That was easy.

The problem I had was in converting the defensive runs to wins to add/subtract to the pitcher’s efforts. The runs I take away from defense are not leveraged, but these wins are already leveraged. I feel like I’m adding/subtracting apples from oranges, so to speak.

Marlin Maniac, a Florida Marlins blog
Come attend Intro to Sabermetrics 101!
Check me out at Beyond the Box Score as well.

by SFiercex4 on Jan 4, 2010 12:43 PM EST reply actions

I think I might be able to rerun the numbers with your adjustments,

Especially if you can send me the results for step 1.

It would be sum(WPA/LI * ceLI) + DEF * ceLI where ceLI = chained entry LI. Does that make sense?

by Dan Turkenkopf on Jan 4, 2010 1:20 PM EST up reply actions

Yeah, I guess that makes sense.

For Step 1, the other primary issue is runs → wins conversion. How many runs / win would we use, given that we generally adjust pitcher runs/win based on their run allowance? I was initially thinking about plugging it into the converter FanGraphs uses (shown here). It’s either that or use Pythagenpat, but I think that’s complicated.

If anyone comes up with something good for this, I’ll look into some numbers this week.

Marlin Maniac, a Florida Marlins blog
Come attend Intro to Sabermetrics 101!
Check me out at Beyond the Box Score as well.

by SFiercex4 on Jan 4, 2010 4:50 PM EST up reply actions

I use a custom run environment based on park / year to calculate my WPA

so that’s probably the right value to use for League RA, right?

by Dan Turkenkopf on Jan 4, 2010 5:12 PM EST up reply actions

For the league RA, I believe that works.

My worry is what to use for the pitcher RA. Is it RA/9 including the defensive runs? I’m not sure about this part.

Once you get that, you can convert those defensive runs into wins and leverage them as you mentioned at the top.

Marlin Maniac, a Florida Marlins blog
Come attend Intro to Sabermetrics 101!
Check me out at Beyond the Box Score as well.

by SFiercex4 on Jan 4, 2010 5:42 PM EST up reply actions

I think that might work

RA/9 is theoretically pitching + fielding. If you correct for fielding, then you’re left with pitching.

But RA/9 is going to be misleading for relievers because of things like inherited runners and leaving with runners on base. It’ll be less misleading for closers because they tend to start/end innings.

Maybe something like BaseRuns/9?

by Dan Turkenkopf on Jan 4, 2010 9:22 PM EST reply actions