UZR's (and Most Other Defensive Metrics) Limitation in Year to Year Analysis
by Jeff Zimmerman (TucsonRoyal) on Aug 15, 2009 5:07 PM EDT
24 comments
Defensive metrics have been evolving for years. One of the major uses for defensive metrics is see how many runs a player saves compared to other players in a season. This value is added to the player's offensive contributions and the player's total contribution to his team is measured.
A very familiar measure of this overall contribution is Wins Above Average (WAR) which is track by Fangraphs. The defensive metric that Fangraphs uses in its WAR calculations is UZR (Ultimate Zone Rating). The measure is considered one, if not the best defensive measure available today.
Note: My critique is on UZR, but almost any other defensive metric available today adjusts to a yearly average. I will be using UZR for my example (mainly because I have done quite a bit of work with it and can reference the work), but the writeup applies to any other defensive metric that adjusts to a yearly value.
The problem with UZR (and almost all other defensive metrics) is that they are adjusted each year to an average value. I have seen people, including myself (here and here and here and here), use year to year values of UZR to help predict a player's defensive ability or to show their defensive ability. Here are two examples (one theoretical and a real world one) of how a players UZR can be effected.
Example 1: Say in 2008 all the shortstops stayed the same as the ones that played in 2007, except the top four defensively rated shortstops got hurt and missed the year. They were all replaced by below average defensive shortstops. If a shortstop that was league average defensively in 2007, he would now be above league average even if he and all the other shortstops that played exactly the same in 2008 as in 2007 because of the 4 that missed the entire season.
Example 2. In 2005, the formulas that are used to calculate UZR loved the effort Orlando Cabrera displayed, especially with the Range Factor component of the UZR values. The following chart is Orlando's Range Value and Total UZR for 2004 to 2006.
| Range | UZR | |
| 2004 | -6.8 | -2.3 |
| 2005 | 16.3 | 20.1 |
| 2006 | -3.7 | -0.8 |
As you can see Mr. Cabrera's UZR value jumped by 22 points in 2005 and in 2006 it went down to 2004 levels. Want to guess what the other shortstops UZR values did in 2005. Take a look at a chart of the drop off.
As it can be seen in 2006 when Cabrera's numbers lowered, everyone else's UZR values adjusted up close to about what they previously were at.
Besides the two previous examples of a player getting hurt or a player playing significantly better, here is a list of other possible causes:
-
Small sample size - Unlike offensive stats, defensive stats get broken down into 7 to 9 (depending on the defensive metric if catchers and pitchers are measured) additional categories. This small sample size lends itself to quite a bit of variability.
-
A player can change positions - This year, Coco Crisp came to the Royals and played center field. This moved David Dejesus over to left field. David was an average to above-average center fielder (lifetime value of 4.6 UZR/150games), but he is a great left fielder (lifetime value of 18.4 UZR/150 games). I would be surprised if the UZR values for left field are down this year and it can all be blamed on Dejesus.
So what are some possible ways to get a year to year UZR comparable values
-
If you are looking at 1 player, look at 5-6 other players at that position and see if they have the same decrease or increase to player you are looking at has.
-
Compare the player's UZR numbers to Tom Tango's Defensive Scouting Report .
-
Find a metric that doesn't adjust to yearly values such as Range Factor. Though Range Factor is not a good method, it might allow you to see if the player was constant or not in the year in question. I have asked around and none of the complex defensive metrics don't adjust yearly. If any reader knows of any metric that does adjust to a yearly value please share.
I hoped to increase the understanding of the limitations of using UZR (and other defensive metrics) when comparing year to year values. As always your comments and suggestions are welcome.
3 recs |
24 comments
|
Email
Print
Comments
Good stuff Jeff
I remember someone on FanGraphs (believe it was Eric Seidman) mentioned a simialr concern like this a bit before. It’s definitely a limitation to try and tell what is noise/variations in the average and what’s talent, let alone what’s luck.
In my case, I always try to compare with Tango’s Fan Scouting Report for the scouting side of defense. It’s not always easy (how much better better is this guy than that guy, and how much of a year-to-year difference is based on bias from the previous year), but it tends to agree with my opinions, so I like it.
by SFiercex4 on Aug 15, 2009 5:18 PM EDT reply actions 0 recs
That blog entry is here. Check the comments – I posted a look at the year-to-year change of UZR for shortstops and found a pretty insubstantial change (about one run per year in UZR/150). I don’t think this is a meaningful difference.
And please, for the love of God, never use Range Factor. Ever ever ever ever ever. Ever.
Ever.
by cwyers on Aug 15, 2009 5:49 PM EDT up reply actions 0 recs
That is a great entry -- how did I miss it.
It is exactly what I was wanting to come across here with.
I had to use the injury example in 2 thread comments in the last couple week here so I wanted an article I could reference when people brought it up. Now I have 2.
Jeff Zimmerman - Protecting the world from RBI's and Wins from my mom's guest house.
by Jeff Zimmerman (TucsonRoyal) on Aug 15, 2009 6:04 PM EDT up reply actions 0 recs
See this and the comments thread for those wondering about range factor.
http://www.beyondtheboxscore.com/2009/8/4/976512/graph-of-the-day-range-factor
---
http://www.beyondtheboxscore.com
http://www.rightfieldbleachers.com \ twitter
by Jack Moore on Aug 15, 2009 6:10 PM EDT up reply actions 0 recs
Great stuff
Probably something that should be obvious, but something I had never thought of. Thanks for spelling it out so succinctly, with pertinent examples.
The Crawfishboxes
A good friend of mine used to say, "This is a very simple game. You throw the ball, you catch the ball, you hit the ball. Sometimes you win, sometimes you lose, sometimes it rains." Think about that for a while.
by DyingQuail on Aug 15, 2009 5:38 PM EDT reply actions 0 recs
I understand where this is coming from.
I think, however, that the limitations of using UZR yearly isn’t coming from the changes in yearly averages. Basically, if you stay constant while the rest of the league gets better, you’re less valuable, aren’t you? Sure, your fielding ability is still the same on an absolute scale, but what matters to us is how you compare to other guys.
Similarly, if I’m in a .200 wOBA league, and I’m putting up a .250 wOBA, I would look pretty awful by pure wOBA, but by wRAA I’d look pretty damn good.
---
http://www.beyondtheboxscore.com
http://www.rightfieldbleachers.com \\ twitter
by Jack Moore on Aug 15, 2009 5:55 PM EDT reply actions 0 recs
This was exactly what I was thinking
WAR also adjusts for the league average wOBA, so for example if Pujols Hanley and Mauer get hurt it would have some effect on the league wOBA, though obviously less so.
Dating girls is like starting pitching depth, you think you have a good full rotation, even too many starters, then in an instant as soon as you trade your depth away injuries decimate your rotation and you are forced to start Sidney Ponson.
by designatedforassignment on Aug 16, 2009 10:55 PM EDT up reply actions 0 recs
the article
I’ll say this for now, for whatever it is worth:
I could change the (computer) code to NOT zero out UZR per year in about 3 seconds. I don’t do that because I found that if I do that, it introduces other problems, mainly because the data is not consistent from year to year, at least it wasn’t. It is probably 6 of one and half a dozen of the other. So, let’s not call this a “flaw” in the system. It is a relatively simply matter for anyone to look at the change in talent from one year to the other and make adjustments.
Most importantly, what exactly do you want the baseline to be? It has to be something. In other words, UZR is ALWAYS expressed as runs saved or cost as compared to an average fielder at that position. That “average” could be 1 year, 3 years, 5 years, etc. It is not like there is an absolute number that is not compared to a baseline. That isn’t the case. While the sample size is indeed larger for offensive metrics, since the comparison is usually every player in the major leagues, but offensive metrics like OPS+ and linear weights (and offspring of lwts) are relative to league average for that year as well. We live with that even though we recognize that overall offense for eacgh league or both leagues can change from year to year.
by mgl on Aug 15, 2009 8:34 PM EDT reply actions 0 recs
I think his point is that people don't do this when calculating WAR
It is a relatively simply matter for anyone to look at the change in talent from one year to the other and make adjustments.
This has long been a contention of mine when using UZR to value a player in a calculation like WAR: We’re simply calculating what the system can calculate, and the system isn’t meant to determine player value. That being the case, we need to be looking at changes in talent, the sample sizes, and so on before we make these calculations. I believe the “average” for wOBA is different and adjusted for each league in each year, so UZR data should be reported the same way.
"I just wish that the late Harry Caray were still around so I could hear him mispronounce 'Kosuke Fukudome' every fukun' night" -- Dennis Miller
by fourstick on Aug 17, 2009 11:17 AM EDT up reply actions 0 recs
UZR is reported that way..
A 0 UZR is exactly average for the given year, every year.
The average wOBA changes by year. This would be more comparable to something like OPS+.
A 100 OPS+ is exactly average for any year, every year. A player in 1910 with an OPS+ of 100 may have had an OPS of .750 while a player in 1998 with an OPS+ of 100 may have had an OPS of .850.
The issues being raised above are that UZR provides a comparison to average, but average changes every year so that it is not a good measurement of an individual’s skillset — just like saying Player A had an OPS+ of 105 last year, and an OPS+ of 95 this year. That player could have done the exact same thing, except there were better players in the league.
by Trickman on Aug 17, 2009 2:23 PM EDT up reply actions 0 recs
Except we can see what the change in average OPS easily.
And we can tell if the batter is worse than last year, or if the league is better.
@bs_uf15bosox9be The Original Gameday; Learn to use SB Nation
by bs.uf15bosox9bears23 on Aug 17, 2009 10:15 PM EDT up reply actions 0 recs
Maybe publish the change from year to year?
@bs_uf15bosox9be The Original Gameday; Learn to use SB Nation
by bs.uf15bosox9bears23 on Aug 17, 2009 10:13 PM EDT up reply actions 0 recs
follow-up
“I think, however, that the limitations of using UZR yearly isn’t coming from the changes in yearly averages. Basically, if you stay constant while the rest of the league gets better, you’re less valuable, aren’t you? Sure, your fielding ability is still the same on an absolute scale, but what matters to us is how you compare to other guys.”
That is an excellent point. If UZR is billed as "runs saved or cost as compared to an average fielder at that position (for that time period), then you are getting exactly that. Where is the problem?
It does become somewhat problematic when you combine years, but, as I said, that can be adjusted for.
by mgl on Aug 15, 2009 8:59 PM EDT reply actions 0 recs
I do single year averages
For all the reasons mentioned above. Plus, you aren’t seeing the same group of hitters every year. Not that big a deal if you’re comparing 2006 to 2007, but for range factor or Totalzone, going back into the late 80’s, you’d have some big differences, at least I think. There was a leaguewide jump in BABIP between 1992 and 1994, and we’ve stayed at the higher level. This might or might not affect UZR, which could show harder hit balls and explain it, but Totalzone will track well with team DER – you could not compare fielders without establishing a league baseline.
I have explained in various posts how I calculate totalzone. I think anyone could duplicate it provided they know how to work with retrosheet files and have about 200 or so hours to kill. And if you want to set constant averages for every position and not do year by year, be my guest.
The HK-47 hitting droid is the finest line drive machine ever built
by RallyMonkey5 on Aug 16, 2009 11:01 AM EDT reply actions 0 recs
What I see is people being able to determine WHY a player's defensive rating changed.
Say you are a Jeter fan and see that his UZR went from 0 to – 15 from 2004 to 2005. Without doing a 15 minute study of various other factor it is almost impossible to determine if the drop is because of Jeter or because of changes in the talent pool at SS.
Now with a hitter, you see his OPS+ drop dramatically, you can look at his other readily available stats and see if it was him declining or everyone else getting better in less than a minute.
Finally, we could see huge jumps in OPS+ if it was divided into 7 categories and players were moved around in these categories.
Jeff Zimmerman - Protecting the world from RBI's and Wins from my mom's guest house.
by Jeff Zimmerman (TucsonRoyal) on Aug 16, 2009 12:03 PM EDT reply actions 0 recs
one object lesson here might be
The influx of a lot of rookie CF this year: rasmus, mccutchen, morgan(who has moved over to cf recently), and I’m blanking on at least one other name. Iirc, the defensive stats of some older cf have dropped further (preinjury beltran sticks in my head); aging is also a factor.
the truth can't hurt you, it's just like the dark/ it scares you witless, but in time you see things clear and stark -- macmanus
by tom s. on Aug 16, 2009 10:55 PM EDT via mobile reply actions 0 recs
Franklin Gutierrez
would be the guy you’re missing. Not a rookie, but he moved from playing in the corners in Cleveland to playing CF in Seattle this year.
"I just wish that the late Harry Caray were still around so I could hear him mispronounce 'Kosuke Fukudome' every fukun' night" -- Dennis Miller
by fourstick on Aug 17, 2009 11:21 AM EDT up reply actions 0 recs
Dexter Fowler
is the rookie you are missing
by bballrox4717 on Aug 17, 2009 2:04 PM EDT up reply actions 0 recs
Could their be a way to show both?
For Rally and MGL, could you not display the number of runs saved against a fixed value AND against the average?
F*** Billy Beane... actually, I kinda like Holliday
by vivaelpujols on Aug 17, 2009 6:19 AM EDT reply actions 1 recs
That fixed value is far more arbitrary than an average.
What’s the point of looking at it in comparison to any fixed value? It’s telling you the exact same thing. We seem to be jumping through hoops here to try to show something in a simpler way when it just isn’t there.
---
http://www.beyondtheboxscore.com
http://www.rightfieldbleachers.com \ twitter
by Jack Moore on Aug 17, 2009 11:45 AM EDT up reply actions 0 recs
The fixed value isn't supposed to mean anything, just to be a reference point
F*** Billy Beane... actually, I kinda like Holliday
by vivaelpujols on Aug 17, 2009 12:16 PM EDT up reply actions 0 recs
I'm not convinced a single-year baseline is what we want for value, either.
I mean, defense includes position-specific numbers plus a position adjustment. If there’s an influx of good-fielding SS, that should both change the SS baseline, but also the SS position adjustment. So unless we’re doing yearly position adjustments (which might be the right way to go except for issues with not having enough data) then using multi-year averages for both pieces will be more accurate when combined.
Beyond the Boxscore Not a member? Sign up.
by Sky Kalkman on Aug 17, 2009 12:13 PM EDT reply actions 0 recs
I would love to see UZR values baselined to when they back.
You would be able to see if a position is getting more talent.
You would see where a player stand now compared all time.
Larger sample size.
Why is having more information a bad thing?
Jeff Zimmerman - Protecting the world from RBI's and Wins from my mom's guest house.
by Jeff Zimmerman (TucsonRoyal) on Aug 17, 2009 12:44 PM EDT up reply actions 0 recs
Agreed.
I think something like this with a fixed evaluation comparison would be helpful in predicting an actual player’s contributions.
The value would not be meaningful to WAR, but it would be more useful in predicting future year values. Having a fixed “average” and then using that to project the average for the following year based on the new sample of players for the position would seem to be a much better predictor of year-to-year defensive ability than the current UZR.
by Trickman on Aug 17, 2009 2:34 PM EDT up reply actions 0 recs
Comments For This Post Are Closed
Secondary Sidebar
User Tools
FanPosts
Community blog posts and discussion.
BtB on Facebook
Certain photos copyright © 2009 by Associated Press or Getty Images. Any commercial use or distribution without the express written consent of Associated Press and Getty Images is strictly prohibited.
