Quantifying the Impact of Defensive Uncertainty

Recently in the sabermetric community there has been a lot of discussion about fielding stats and their inclusion in WAR (see for example this thread, or this one at The Book blog) given the uncertainty behind the data (batted ball type, hit location etc.). With that in mind I thought it would be an interesting exercise to see how applying uncertainty to the defensive runs above average (DRAA) numbers affects the 2009 fWAR leaderboard. My method for applying the uncertainty is pretty simple; I just ran a Monte Carlo simulation using a normal distribution for the simulated DRAA with a mean of the DRAA reported by Fangraphs and a standard deviation of 5 runs. The following table looks at how often the top 10 players in fWAR fell into each of the top 10 slots after running the simulation 10000 times.

 

1 2 3 4 5 6 7 8 9 10
Albert Pujols 62% 23% 9% 4% 1% 0% 0% 0% 0% 0%
Ben Zobrist 22% 36% 22% 11% 5% 2% 1% 1% 0% 0%
Joe Mauer 12% 24% 29% 17% 9% 5% 2% 1% 0% 0%
Chase Utley 3% 8% 16% 23% 19% 13% 9% 5% 3% 1%
Derek Jeter 1% 4% 10% 16% 18% 17% 13% 9% 5% 3%
Hanley Ramirez 0% 2% 7% 12% 16% 18% 16% 11% 8% 5%
Evan Longoria 0% 2% 4% 10% 14% 17% 17% 14% 9% 6%
Prince Fielder 0% 0% 2% 5% 8% 12% 15% 16% 15% 10%
Ryan Zimmerman 0% 0% 1% 2% 4% 7% 11% 15% 16% 15%
Adrian Gonzalez 0% 0% 0% 1% 3% 6% 8% 13% 16% 15%

 

So if you buy my 5 run SD assumption then the impact on ordinal ranking is the above.  Clearly the impact on overall WAR (and thus $/WAR) isn't captured in the above analysis.

 

This is just a quick look at the subject, but I think there may be more to uncover like looking at different fielding metrics in place of UZR.  Either way it answered one of my questions, "What orders of magnitude are we talking about?"

Update:  Here's the same table with a SD of 10 runs

1 2 3 4 5 6 7 8 9 10
Albert Pujols 38% 21% 14% 9% 6% 4% 3% 2% 1% 1%
Ben Zobrist 22% 20% 16% 11% 9% 6% 4% 3% 2% 2%
Joe Mauer 15% 17% 15% 12% 10% 8% 6% 4% 3% 2%
Chase Utley 8% 11% 12% 11% 10% 9% 8% 6% 5% 4%
Derek Jeter 5% 8% 10% 10% 10% 9% 8% 7% 6% 5%
Hanley Ramirez 4% 7% 8% 9% 10% 9% 8% 7% 6% 5%
Evan Longoria 3% 6% 7% 9% 9% 9% 8% 7% 7% 5%
Prince Fielder 2% 4% 5% 7% 7% 8% 8% 7% 7% 6%
Ryan Zimmerman 1% 2% 4% 5% 6% 7% 7% 7% 7% 6%
Adrian Gonzalez 1% 2% 3% 5% 5% 6% 6% 7% 6% 6%
X
Log In Sign Up

forgot?
Log In Sign Up

Please choose a new SB Nation username and password

As part of the new SB Nation launch, prior users will need to choose a permanent username, along with a new password.

Your username will be used to login to SB Nation going forward.

I already have a Vox Media account!

Verify Vox Media account

Please login to your Vox Media account. This account will be linked to your previously existing Eater account.

Please choose a new SB Nation username and password

As part of the new SB Nation launch, prior MT authors will need to choose a new username and password.

Your username will be used to login to SB Nation going forward.

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

Join Beyond the Box Score

You must be a member of Beyond the Box Score to participate.

We have our own Community Guidelines at Beyond the Box Score. You should read them.

Join Beyond the Box Score

You must be a member of Beyond the Box Score to participate.

We have our own Community Guidelines at Beyond the Box Score. You should read them.

Spinner.vc97ec6e

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_9351_tracker