Value, Context, and Clutch: a Conversation

Matt: Hey Spencer, let’s have a debate. I think batters should get more credit for hitting a double with a runner on first than with the bases empty. Respond.

Spencer: Why? Is it a skill to hit with runners on?

Matt: You should get credit for hitting well in higher leverage situations, but adjust for opportunity.

Spencer: But is it a skill?

Matt: Doesn't matter.

Spencer: Why? We're evaluating a player's skill.

Matt: We’re measuring value, not skill.

Spencer: Skill=Value

Matt: I disagree. A grand slam has much more value than a solo home run, even though they require about the same amount of skill.

Spencer: Yes, but the individual player didn't do anything extra on the grand slam than the home run. Why does he deserve credit for the guys in front of him getting on?

Matt: I’ll answer with an analogy. Say 2 batters had two PAs during the game, one with no one on, and one with the bases loaded. Player A hits a home run with no one on and Ks with bases loaded. Player B does the opposite.

Spencer: They should be credited equally.

Matt: But player B did a better job. They had the same opportunity, and B came through when it mattered more.

Spencer: Why does player B get credit for having guys on when he hit his HR instead of striking out?

Matt: I'll turn that Q around: why shouldn't B get credit for hitting a home run when it mattered more?

Spencer: Because they both did the same thing. They both hit a home run and then struck out. One was just lucky with the sequencing. Player B had a larger impact on the game, but he wasn't more valuable.

Matt: He contributed more to the end result.

Spencer: No, player B, the baserunners, and the sequencing of the events contributed more to the end result. They both did the same thing individually. This is just the RBI argument.

Matt: But they had the same opportunity. As I see it, there are two arguments against RBI: one is opportunity. That is, some players just get more RBI chances. The other is your argument that context shouldn’t influence value, which I disagree with. I'm not saying you should reward players for being in more RBI situations. I'm saying you should reward players for coming through when those RBI situations come up. As in, use WPA/LI instead of WPA (except that WPA is flawed).

Spencer: But when evaluating the two players we can't credit one extra when they both did the same thing individually. Player A hit a 400 foot home run to center field. Player B hit a 400 foot home run to center field. Player A had a runner on first. Player B had bases loaded. What does that second part add to the conversation of what player A and B did?

Matt: Ok just taking those two cases completely independently of everything else, they have the same "value".

Spencer: Well define value.

Matt: That’s a million dollar question. How about "contribution to the team".

Spencer: Individual contribution to the team?

Matt: Sure.

Spencer: Then you can't be context-dependent.

Matt: Why?

Spencer: Because breaking it down to the simplest level possible, if a player A has the same batted ball chart as a player B, so they hit the same hits, isn't that what they contributed? Your goal is to succeed in your at-bat. If every player gets on base, then the team wins.

Matt: But context matters. The goal for the players was to help their team win, not boost their stats. B helped his team win more, and A is thinking, "I wish I had hit a home run with the bases loaded instead of bases empty".

Ok let’s try another analogy. In my job, I test software for bugs. Say I test a program that has 100 bugs and I do a great job at it. I find 99 of the 100 bugs. But they are really minor bugs, like typos and minor aesthetic issues - they don't really matter. However, as it turns out, the 100^th bug, the one I missed, was really important. It ends up creating a mess and seriously harming the customer.

Spencer: Then you screwed up that one bug, but you should still get credit for the other 99.

Matt: Yes, but that one bug was more important.

Spencer: They are all independent of each other - they’re separate bugs.

Matt: Ok, say someone else doesn't do as well with the 99 minor bugs. He only gets like 70 of them, misses all these typos and minor issues with the software. BUT he catches the huge one, and in doing so saves lives. That guy did a much better job than me because that one bug was much more important.

Spencer: No he didn't – you did a better job.

Matt: We each had the same bugs to find, and even though I caught more of them, the other guy did a better job because he got the big one.

Spencer: You fixed 99% of them.

Matt: So? They were minor bugs.

Spencer: Well wait a second, was the last bug harder to find?

Matt: It wasn't necessarily harder to find, but it mattered a lot more. So maybe going forward I'm still the better tester - I find more bugs. But if we're evaluating performance, the other guy gets the raise, not me.

Spencer: But the 100th bug was of equal skill then - you did better because you fixed 99% of them.

Matt: Tell that to the people who died because I didn't find the bug.

Spencer: It’s bad luck with sequencing. Well I mean it has nothing to do with your skill

Matt: It wasn't just random though.

Spencer: Then you're saying that the 100th bug was more difficult.

Matt: No.

Spencer: What was different about that bug?

Matt: Let's say it was in a different place in the software, a place I usually check, but I just blanked and didn't check it.

Spencer: Is it a trend that you miss this 100th bug?

Matt: No. I'll probably check that place the next 1000 times. But I didn't that one time, and it screwed everything up.

Here, I've got another analogy: you're a track and field runner; you run the mile. Say your true talent remains constant throughout the season - but sometimes you're going to have better and worse days. So your true talent is, say, a 5:00 mile, but some days you run 4:55, some you run 5:05, etc.

So your mile times follow a normal distribution basically, but that doesn't mean that you just got lucky when you ran a good mile. It wasn't just random variation - it was YOU that ran the mile. YOU get credit for it. So in the final meet, you have your best mile time ever at 4:40, your "true talent" didn't change - you're still going to average 5:00 miles going forward - but you damn well deserve praise and recognition for running that fast.

Spencer: But when I'm evaluating myself as a runner I'm looking at what I did individually. I would credit myself with running that fast. It would be awesome. But I don't see how this relates with what we're arguing.

Matt: It’s related. What I'm saying is that even if something has random variation, that doesn't mean the person shouldn’t receive credit or blame for going above or below the mean.

Spencer: How does that support your argument?

Matt: In the bug testing example, my "true talent" is to catch 99% of bugs. As it turns out, that 1% happened while I was testing the most important bug, but that doesn't mean it wasn't my fault.

Spencer: But it’s not your fault. You did a great job – you got 99% of the bugs.

Matt: No, just like the running example, you should get credit or blame for going above or below your mean time. You don't say, "he just happened to run a 4:40 mile in the championship meet".

Spencer: Well he did. He got lucky. Unless he did something different. Did he do something different, knowing the championship was more important?

Matt: He ran a faster mile.

Spencer: Did he adjust his shoes for the track? Because he knew it was different?

Matt: Does it matter?

Spencer: Yes, because otherwise it’s just randomness.

Matt: I mean he clearly did something different if he ran faster. Maybe not consciously, and maybe not because it was the championship, but he did something different, and it made him go faster - he should get credit for that.

Spencer: So perhaps a player subconsciously hits better in big spots. Then that is a skill and he should be credited for it; but otherwise it’s just randomness.

Matt: Well, not "hits" better, "hit" better. Maybe it was a one-time thing.

Spencer: Then its randomness.

Matt: See that is the jump I disagree with. That's the fundamental thing we're arguing about, that if it's not a sustainable skill, it's randomness.

Spencer: If Johan Santana hits a home run against the Reds in June of 2010, did he do something different that allowed that to happen, or is it just dumb luck?

Matt: Maybe it's dumb luck. Maybe. I still want to reward a player for coming through in the clutch. I'll take the risk that it's all just dumb luck.

Spencer: I still don't see why we should.

Matt: For the same reason we reward the runner. We assume that the events that happen are at least somewhat under their control.

Spencer: Yes the individual at-bat is.

Matt: If I'm the parent of that runner, I'm proud of my son/daughter for running that fast, EVEN if it was just random variation.

Spencer: Well then what is random variation?

Matt: Ok maybe random variation was the wrong choice of words.

Spencer: But that’s what the sequencing is. It’s random.

Matt: Even if the data points turn out in a completely random pattern, that doesn't mean that complete randomness is the reason for that pattern. Does that make any sense?

Spencer: I get what you're saying but I disagree with it. It is random. If you control for all other factors, the player who did the best in their individual at-bats is the best player.

Matt: But that is also subject to random variation. Take a completely average true talent player. If we simulate his season 100,000 times, he'll have a couple MVP seasons. Do we give him credit for that MVP season even if it was a product of random variation? I think we do.

Spencer: When evaluating that season yes. But if we are evaluating his body of work it should be regarded as an outlier, an individual event, separate from the rest.

Matt: Of course, but we still give him credit, even though it's random variation. So if we can give the player credit for an awesome season even though it was a product of random variation, why can't we give a player credit for doing well in clutch situations even if it's random variation?

Spencer: I'll give him credit for that season, because it happened, but I don't see how that proves that we should give a guy credit for a hit with runners on as opposed to one without runners on.

Matt: Because the one with runners on mattered more. He came through in the clutch. He added more runs given opportunity than the other guy. Even though it was an outlier and not repeatable, it happened, just like the MVP season from the average player.