In analyses, we tend to focus on the process. But how much do we really know about the process anyway? Should we be paying more attention to results?
For the sake of clarity, I’m going to paint an extreme hypothetical scenario here. Say that a given team has two outs in the ninth inning of a game. They’re down by one run, with runners on second and third. There’s a right-handed pitcher on the mound. One of the pinch-hit options is the right-handed Ryan Braun. The other is the switch-hitting Emmanuel Burriss.
With the game on the line, the manager decides to insert Burriss as the pinch hitter because of the platoon advantage. Braun is the superior hitter, even against right-handed pitchers, and it’s not even close. No manager in his right mind would ever actually do this, but again, this is merely hypothetical.
Now, let’s say Burriss comes through in the situation with a game-winning two-run single. I said this was an extreme example, and crazy as it sounds, it’s within the realm of possibility that Burriss could collect a game-winning hit. Would that justify the decision to use him as a pinch hitter? Of course not. Regardless of the results, the process was stupid.
After the game, we’d all say that the team won in spite of the manager, not because of him.
The focus, in analysis, tends to be on process. As it should be. Results are secondary. This is how we go about judging lots of things in baseball -- managerial decisions, transactions led by a front office, etc.
Here’s another situation, only this one isn’t hypothetical: in January, the Tampa Bay Rays signed Fernando Rodney to a one-year contract (with a club option for 2013) worth $2M in guaranteed cash. The collective reaction, on Twitter and whatnot -- at the risk of generalizing -- was one of derision. Rodney was set to turn 35 years old in March. He was coming off a season in which his walk rate had actually been higher than his strikeout rate.
He hadn’t posted a sub-4.00 FIP since 2007, and he hadn’t posted a sub-4.00 ERA since 2006. Not once, in his ten-year career, had he maintained an above-average walk rate. Two million dollars isn’t exactly a lot of money, but it seemed awfully optimistic of the Rays to expect Rodney to be a competent late-inning reliever.
Yeah, it seemed that way. Then he went on to post the lowest ERA (0.61) by any pitcher who had ever thrown 50-plus innings in a single season. He had his fair share of help from the Rays’ defense -- particularly expert pitch-framer Jose Molina -- and his FIP was roughly 1.5 runs higher, but Rodney was spectacular nonetheless. He was so good, in fact, that his ERA+ of 634 was higher than the sum of his individual ERA+ marks from each of the previous six seasons.
Did the Rays know what they were doing when they tossed chump change in Rodney’s direction? Surely they didn’t expect historic numbers like this. But there’s a hint that the Rays had something specific in mind when they signed Rodney. They shifted Rodney’s position on the rubber, which altered his release points. And the results were phenomenal.
I didn’t think much of the deal at the time, but wouldn’t it be silly to look back on it and say it was a foolish signing, in light of the results? The process didn’t look good, necessarily, but here’s the thing -- we didn’t really have a firm grasp of the process. We still don’t. But we know that the Rays had Rodney make some adjustments right from the start, and we know that these adjustments paid off. The results give us some inkling of an idea what Tampa Bay was thinking when they signed Rodney.
This tendency to focus on the process, to analyze based solely on statistical projections and the like -- because that’s the data available to us, obscures the fact that we know very little. There was something similar discussed on the Baseball Prospectus Up & In Podcast a while ago that resonated with me. Thankfully, it was transcribed by mightyflynn on Tumblr:
PP: They’re dealing with metrics that people in the public sphere don’t have access to and can’t—and aren’t writing about. It’s a completely different story.
KG: And would kill to see.
PP: Right. They are dealing with different information.
KG: Yeah, they have data people would love to have public. They have all sorts of stuff, they’re doing amazing stuff with it. They’re doing fantastic, ridiculous stuff that we would love to have public and it would be awesome. And, you know, we’ll continue to try to catch up to em, but they have certain advantages on data and people, and they’re doing some ridiculously cool stuff that no one talks about because it’s all private to the club, for good reason. It’s part of their perceived competitive advantage.
There’s a wealth of data out there -- on Fangraphs, Baseball Reference, et al. -- and I’d hate to understate its value. The scope of information freely available to the public is astounding. But it pales in comparison to what front offices have. They have a much broader understanding of all of this, whether through scouting or field f/x technologies or other various sources. And we'd be arrogant and ignorant to think otherwise.
We owe it to ourselves to realize and recognize just how little we know. This means giving teams the benefit of the doubt. This means approaching analyses with a level of calculated uncertainty. And this means keeping an open mind. We can’t dismiss results. We have to weigh them, examine them, and seek to learn something from them; there is, after all, a lot to be learned from them.