Let’s break down UZR for a second. The essence of any fielding metric relative to average is:

Outs made – Expected outs

That gives you plays above average. Most metrics convert this to runs above average, although for our purposes here the nature of this conversion is unimportant. (It generally works out to multiplying plays by a constant, somewhere around .7 to .8. There are more complicated methods, and dependent upon how reliant they are on the particular batted ball data they may introduce additional “noise” or “bias,” but for now let’s pass that on by.)

There is very little bias in the “observed outs” term – there is a little, though. Mostly this is because of borderline GB/LD – some metrics exclude LD plays for infielders, and if you have a park effect on the GB/LD boundary (and I think you do) that’s going to complicate matters. And there’s practically no random measurement error on observed outs.

Most of our interest is in bias on expected outs, though. And here’s where it gets interesting. Expected outs comes from looking at the data, and saying “given these parameters, what’s the average rate of plays made?” And common to most of these metrics are the parameters of batted ball type, distance and angle (typically broken down into “zones” – STATS zones, for instance, run roughly 4.1 degrees). So your expected outs category is driven by:

• The actual distribution of BIP.

• Any systemic biases you have – observer positioning biases, range biases, etc.

• Random measurement error in the batted ball distribution.

So in terms of what HAPPENED (as opposed to trying to forecast events), almost all of your error is centered on the expected outs term.

When you regress, you do so based upon the expectation that if you pick the top 20 percent of players, for instance, collectively they will have gotten lucky in terms of random error. Is this true?

We don’t know. It’s possible, for instance, that the top 20 percent may have gotten “lucky” in terms of actual outs made given their BIP distribution, but were average in terms of random error. Or perhaps their “luck” was to play in a park that encourages scorers to assume they had harder BIP to field than the average fielders.

It is very difficult to know how much regression to do without having some understanding of how much the other variables affect expected outs. And simply regressing UZR, rather than regressing the expected outs component, makes the process much more difficult.