Sloan Reaction: The Automated Prospect Model

Like fellow BtBS writer Blake Murphy, I also attended the 2013 MIT Sloan Sports Analytics Conference held in Boston. Blake noted this in his initial reflections piece, but many of the conference's speakers and panels focused on topics that did not specifically involve baseball. Still, it was a fascinating two days and the few baseball-specific presentations that were at the conference were excellent.

One of the baseball-specific talks in the "Evolution of Sport" series was entitled "The Automated Prospect Model" and it was presented by MLB consultant and former writer for The Hardball Times, Adam Guttridge. In the presentation, Guttridge outlined a system to automate prospect lists with the hope that the model will improve upon the ranking methods that mainstream prospect writers currently use. He cited two major problems with the top-100 lists that Baseball America, ESPN's Keith Law, and MLB's Jonathan Mayo put out each year:

Typically the lists involve the judgments and opinions of many scouting voices added up to form a collaborative sentiment about players.

The lists stop at 100 players because the rankers do not enough have enough confidence to keep going.

Those are two very interesting points for different reasons. I have always felt that the surveying nature of how information is collected for those lists is a limiting factor in their usefulness. There is no doubt that it is important to gather as much information on each prospect as possible, but each extra report collected only tempers the one that said a player would be a superstar or a bust. I like the idea of getting fewer opinions on players; the important part is just getting the right one or two opinions.

Guttridge's second point is one of my other problems with prospect lists -- what is the need to rank players at all? In reality, lists just don't provide any context, a point that I was happy to see Ben Horrow make on Saturday. If someone says that Baseball America ranked Dodgers' OF Yasiel Puig 47^th this year, we really don't know much about him. On the other hand, if that same person says that Puig has a 68 OFP and a 60% chance of reaching that outcome, now we have context about the player's value. When it comes to prospects, grades and scouting reports are much more important than the actual rankings.

The creation of the list

The nature of the Evolution of Sports talks -- brief 20 minute presentations followed by a few questions -- did not allow for a great deal of details regarding the construction of the list. I would say Guttridge only spent a minute or two talking about how he formulated his list, instead using his time to point out advantages with the model. Despite that, he did at least outline the three-step process he used:

• Utilize Minor League statistics to determine each player's present value.
• Project how the players will develop over the next 5 years using Minor League equivalencies and an aging curve.
• Add some subjective information about defense and pedigree to help shape the rankings.

Although I think there are huge flaws in solely using Minor League statistics for evaluation, mainly due to the development-heavy focus of the minors, I can't say much for or against his process without further details into each step.

Example: Bradley and Head

As examples of prospects that his list ranks very differently than the majority of experts, Guttridge used Red Sox outfielder Jackie Bradley Jr. and A's infielder Miles Head. Looking at the table below, you can see the two players had similar (his words, not mine) numbers split between Hi-A and Double-A, with Head producing a little more power and Bradley showing better rates in both walks and strikeouts.

Stats courtesy of Baseball Reference.

Again, with limited knowledge of the details of how the model was constructed, I want to focus on the points where I can give opinions. So let's ignore the fact that Bradley plays a premium position and hit in much more difficult environments last season. Guttridge did make two points I can examine, though. First he said that Miles Head had similar production at the same levels as Jackie Bradley in 2012, despite being nearly a full calendar year younger. Ignoring the difference in leagues, I don't think you can argue that point. But then, he asked the audience a very interesting question:

What do you think is more likely to happen? Jackie Bradley Jr. suddenly develops 25+ home run power, or Miles Head learns to take 70+ walks a season?

There was no doubt in my mind that Guttridge would say that it is more likely that Bradley starts hitting more home runs as he continues to mature and refine his approach at the plate. To my surprise, however, he actually said that it is more likely that Head begins to walk at a higher rate. My limited scouting background tells me that his notion is flawed, but without hard evidence I cannot get too upset with the claim. Further research is necessary to really make a strong case one way or the other.

Overall, I think the concept of an automated prospect list is an intriguing one, as long as the final list includes some context, rather than just ranking players. I don't think we will ever see a change in the mainstream lists because of their popularity, but an automated list supported by good objective and subjective data has a chance to outperform the others in predictive value. I do wish the time slot for the presentation was longer because I have so many questions about the construction and inputs in the model. Unfortunately, though, one of the best and worst parts about a conference like Sloan is coming home with more questions than answers.

Some final random tidbits:

• The complete list ranks 812 prospects, which he noted is the number that he found to be above prospect replacement level this season.
• Retroactively looking back at the 2012 list, Guttridge noted his list ranked both Oscar Taveras and Anthony Rizzo much higher than others. This year's darling is Rangers' pitcher Cody Buckel.
• The Automated Prospect List was awarded best in category for the EoS addresses. It was one of several baseball-specific talks in the "Evolution of Sport" series. Others included Paired Pitching, Moneyball Revisited, and Hitting 'Em Where They Are.