clock menu more-arrow no yes mobile

Filed under:

Prospect's delight: Which top 100 list is best?

Prognosticators have been ranking prospects for at least two decades now. Which one is the best?

Mike Trout drops some dope rhymes
Mike Trout drops some dope rhymes
Harry How

Introduction

Now what you read is not a test, I'm ranking prospect lists

And me, Rstats, and some math are gonna try to lift the mist

-Wonder Mike Trout

There has been some great analysis on how well Baseball America's prospect rankings translate to future success, see my Prospect Analysis page for examples from various writers around the web. Adam Foster at Project Prospect has done some work with this in his industry comparisons. However, BA isn't the only list around. Throughout the years, various rankings have come and gone. Out of all of these, which was the best prospect list?

Method

Check it out, it's the k-e-n-d-the-a-l-l

And the rest is t-a-u

You see, this is the code for the method of the post

And these reasons I'll bring to you

-Big Bank Hammerin' Hank

First, I compiled 166 prospect lists in the era from 1990 to 2010. I used 2010 as a cutoff to allow players four years to blossom in the major leagues. I found as many as I possibly could, knowing I missed many as well*. Once everything was compiled, the question becomes: how do you figure out which one is best? In my initial research into this, I found the book "Who's #1?" which suggests using Kendall's Tau or Spearman's weighted footrule when comparing ranked lists.

*If you have a list that you would like to be included in this, please e-mail me or contact me on Twitter @stealofhome.

Recently, Neil Paine used Spearman's rho on organizational prospect rankings at 538. However, I believe the fact that I have repeated rankings (I gave a rank of 101 to all prospects from the real list that did not appear on the historical lists) disallows me from using this.

I asked Tango the best way to do this and his idea was to create a fantasy draft for each year and allow each list to draft their first available player, then repeat this for each year and each possible order of drafting, with the list that drafted the highest average fWAR as the winner. This is probably the correct way to do this. However, if I waited until I had working code for this scenario, this research would not be completed until prospect lists (as well as me) were but a dim memory. However, if you are capable of putting that code together, I would be very interested in working with you to publish that research.

I landed on using Kendall's Tau for this analysis and will do my best to attempt to describe it. Note: I am not a professionally-trained statistician, so if you are and see any errors I make in my description, please let me know.

Kendall's tau shows how close two columns of rankings are to each other. It gives a range from -1 (meaning they are ranked in the opposite order) to 1 (they are the same). It compares the number of rankings below and above the rankings and takes into account ties. Here is a youtube video describing part of what it does, although this does not look at ties.

Since Kendall's Tau works by comparing one list to another, I had to find the "true" top 100 for each year. I did this by finding all players eligible for prospect lists each year and ranking them by their career fWAR. Then I compared that true top 100 list to the various prospect lists. This is an area that may need cleaning, as I don't have the proper data to do this perfectly.

If the player did not appear on the list in question, they received a ranking of 101. If the list went over 100 players, I only used their first 100 rankings. If a list ranked fewer than 100 players, I compared it with the true list of that length for that year (e.g., MLB's top 50 lists are only compared to the true top 50 prospects eligible that year). One more wrinkle in this analysis is that not all lists claim the same eligibility standards. Most rank anyone who would be a rookie, but there are some variations on that. This may hurt some lists in these places.

This is an interesting exercise in itself to see which prospects were missed and which players were lurking in the lower levels before exploding on the scene. For instance, Pedro Martinez had the best career of anyone eligible for prospect lists in 1991, but was nowhere to be found on BA's list. However, in 1992, he jumped into the top 10 with a strong performance in three levels of the minor leagues the previous year.

Name True BA
Pedro Martinez 1 101
Chipper Jones 2 49
Mike Mussina 3 19
Jeff Bagwell 4 32
Ivan Rodriguez 5 7
Andy Pettitte 6 101
Jim Thome 7 93
Jim Edmonds 9 101
Mike Piazza 10 101
Kenny Lofton 11 75
Jeff Kent 12 101
Luis Gonzalez 13 101
Brian Giles 14 101
Moises Alou 15 101
Jorge Posada 16 101
Bernie Williams 17 11
Carlos Delgado 18 101
Mariano Rivera 19 101
Chuck Knoblauch 20 72

BA did not have 11 of the top 20 eligible prospects in their 1991 list. However, many of these players were young at the time and did eventually appear on a BA top 100. Bernie Williams was ranked higher (11) than his true score (17). Jim Thome was ranked lower (93) than his true score (7). Kendall's Tau takes all of this into account and spits out a rank correlation of 0.157 - about average for the results I found.

Results

Well it's on and on and on on and on

This post don't stop until I make you yawn

-Master Dillon Gee

Before I get into the results, I want to add a personal note: Please do not use these rankings to trash writers who may rate poorly here. First, I believe this method can be improved upon, which may change the results a bit. Second, and most importantly, prospect analysis is really hard. These lists require hours of research and dedication, making phone calls and meeting scouts, poring over numbers, and doing whatever else it takes to rank players. These are a labor of love for each individual and it takes a lot of guts to put your name next to something that is both outdated almost immediately—thanks to the dynamic nature of young players—and destined to mostly fail-thanks to how hard the majors are.

With that said, here are the raw results after running Kendall's Tau on each of the lists in my database:

Year Length List Tau Tau+
2008 100 Rotowire 0.372 109
2008 100 Mound Talk 0.367 107
2008 100 Mound Talk Community 0.364 107
2008 100 Top Prospect Alert 0.355 104
2008 100 Baseball America 0.352 103
2008 100 Baseball Prospectus 0.348 102
2010 40 The Cardinal Nation 0.340 196
2008 100 ESPN 0.331 97
2007 50 MLB 0.330 137
2007 100 Fantasy Baseball Café 0.325 135
2008 100 Project Prospect 0.320 94
2007 50 Baseball Notebook Fantasy 0.316 131
2010 50 MLB 0.303 175
2002 50 John Sickels 0.297 390
2007 100 Mound Talk 0.289 119
2007 100 Minor League Ball Community 0.281 116
2007 100 FOX 0.278 115
2007 100 MLB 0.272 112
2007 100 Baseball Digest Daily 0.270 112
2007 100 Rotoworld 0.268 111
2007 100 Project Prospect 0.264 109
1997 100 Baseball America 0.263 114
2008 50 MLB 0.262 77
2009 100 Project Prospect 0.261 115
2004 50 CBS 0.260 160
2007 100 CBS 0.257 106
2007 100 Top Prospect Alert 0.256 106
2007 100 Baseball Prospectus 0.255 106
2009 100 ESPN 0.255 112
2001 40 Baseball Prospectus 0.249 185
2007 100 RotoJunkie 0.242 100
2009 100 Top Prospect Alert 0.241 106
2007 100 Baseball America 0.240 99
2005 50 MLB 0.239 139
2005 100 Minors First 0.233 135
2005 100 FOX 0.232 135
2005 100 RotoAmerica 0.229 133
2009 100 MLB Prospect Guide 0.228 100
2006 100 RotoJunkie 0.225 136
2009 50 MLB 0.225 99
2006 100 Rotoworld 0.225 136
2006 100 On Deck 0.222 134
2006 100 SI 0.220 133
2004 100 Rotoworld 0.220 135
2007 75 SI 0.219 90
1991 100 Spring Training 0.218 116
2006 100 RotoAmerica 0.216 130
2009 100 The Hardball Times Fantasy 0.214 94
2004 100 FOX 0.214 131
2010 100 Project Prospect 0.212 122
2006 100 FOX 0.208 126
2009 100 Baseball Prospectus 0.207 91
2005 100 Baseball America 0.206 120
2006 100 Baseball America 0.202 122
2003 50 Baseball Think Factory 0.201 178
2005 100 On Deck 0.200 116
1998 100 Spring Training 0.200 115
1997 100 Spring Training 0.199 86
1995 100 Baseball America 0.199 131
2006 100 FOX Fantasy 0.198 120
1993 100 Baseball America 0.198 134
2004 100 Spring Training 0.195 120
1999 40 Baseball Prospectus 0.194 230
2006 75 Inside the Dugout 0.192 116
2009 100 Baseball America 0.190 83
1999 50 Prospects, P, and Suspects 0.188 224
2004 100 Minors First 0.188 116
2005 100 SportsBlurb 0.187 109
2004 100 Ken Warren 0.185 114
2001 50 Lindy's 0.184 136
2004 100 Baseball America 0.183 113
2010 100 Dobber Baseball 0.180 104
2005 100 Rotoworld 0.179 104
2003 40 Baseball Prospectus 0.175 155
2004 100 On Deck 0.174 107
2007 50 Aaron Gleeman 0.173 71
2005 50 The Hardball Times 0.169 98
2004 100 The Sporting News 0.167 103
2010 100 Baseball America 0.166 96
2010 100 Baseball Intellect 0.164 95
1996 100 Baseball America 0.164 104
1991 100 Baseball America 0.157 84
2006 60 Bob Reed 0.156 94
1996 100 Spring Training 0.152 96
2006 100 Warm October Nights 0.152 92
2003 100 Rotowire 0.150 132
2010 100 AOL 0.148 86
1998 100 Baseball America 0.147 85
2010 100 MLB Prospect Guide 0.146 84
2006 50 The Hardball Times 0.146 88
2005 100 Diamond Futures 0.143 83
2006 50 Sporting News 0.142 85
2001 100 Baseball America 0.141 105
2007 100 Baseball Notebook 0.141 58
2006 100 Baseball Notebook 0.138 83
2005 60 Bob Reed 0.135 79
2006 75 Baseball Analysts 0.135 82
2003 100 Ken Warren 0.134 118
2003 100 Rotoworld 0.133 118
2010 100 Fangraphs 0.132 76
2003 100 Top Prospect Report 0.129 114
2005 100 CBS 0.129 75
2004 50 Rotowire 0.128 79
2003 100 The Sporting News Wheeler 0.124 110
2001 100 Ken Phelps 0.124 92
2003 100 Baseball America 0.123 108
2010 100 The Hardball Times Fantasy 0.122 71
1994 100 Baseball America 0.120 111
2010 100 Top Prospect Alert 0.119 69
2010 100 Baseball Prospectus 0.118 68
2007 100 Sports Weekly 0.116 48
2002 100 Baseball America 0.115 152
2004 100 Diamond Futures 0.111 68
2001 100 Team One 0.110 81
2004 90 Baseball Analysts 0.109 67
2001 100 CBS 0.109 81
2004 50 Baseball Prospectus 0.105 65
2004 50 MLB 0.105 65
1995 100 Spring Training 0.104 69
2001 100 Spring Training 0.104 77
2003 100 Minors First 0.102 90
2001 100 Top Prospect Alert 0.102 75
2010 100 ESPN 0.102 59
2006 100 Diamond Futures 0.101 61
1993 100 Spring Training 0.097 66
1994 100 Spring Training 0.097 89
2003 100 Ken Phelps 0.095 84
2004 50 The Hardball Times 0.094 58
2003 100 The Sporting News 0.093 82
2001 50 John Sickels 0.093 69
2002 100 Strike Three 0.087 114
2002 100 Spring Training 0.086 114
2002 40 Baseball Prospectus 0.086 113
2003 100 Strike Three 0.085 75
2005 50 Baseball Prospectus 0.083 49
2003 100 Prospect Report 0.080 71
2002 100 Top Prospect Alert 0.079 104
2006 50 Baseball Prospectus 0.075 45
1992 100 Spring Training 0.075 105
1999 100 Baseball America 0.074 88
2002 100 Prospect Watch 0.069 91
1992 100 Baseball America 0.067 95
2003 100 Spring Training 0.065 57
2000 100 Baseball America 0.064 255
1999 100 Shane Mills 0.062 73
1999 100 Spring Training 0.059 70
2002 100 Minors First 0.051 67
2002 100 Ken Warren 0.045 59
2000 100 Preview Sports 0.045 -177
2005 75 Baseball Analysts 0.043 25
2007 100 Diamond Futures 0.039 16
1990 100 Baseball America 0.038 139
2006 50 MLB 0.029 -18
2002 100 Ken Phelps 0.028 37
2000 40 Baseball Prospectus 0.024 94
2000 100 Spring Training 0.021 84
2002 100 The Sporting News 0.018 24
2002 100 Rotoworld 0.017 22
1990 100 Spring Training 0.016 61
2000 50 John Sickels 0.011 -45
2000 50 Ultimate 0.011 43
2002 100 MLB Prospects 0.010 13
2003 100 Creative Sports 0.009 -8
1999 100 Fastball 0.009 10
1999 100 CBS 0.004 -4
2000 80 Top Prospect Alert 0.000 2

Congratulations, John Sickels, your 2008 Rotowire list was the best of all time!

The first thing that jumps out from these results is that tau is very highly correlated to the year the list was posted. Since most lists are created from a very similar group of players each year, that is expected. What this really shows, then, is that 2007 and 2008 were good years for prospect lists in general. Because of that, I have also included a corrected tau, which accounts for the yearly average. This is tau divided by yearly tau times 100. This makes... John Sickels' 2002 Top 50 the best list.

This is what the average tau looks like by year:

Prospecttau

There is a cyclical trend here, where the average tau generally increases from 1990 to 1997, decreases to 2000, increases to 2008, and has decreased since then. This has to do with many of the top players graduating into the higher levels of the minors and gaining more national attention. This then creates better lists (e.g., Pedro Martinez going from unranked to the top 10 in 1991).

Finally, this table summarizes these results by list "voice." I have done my best to match these up, but let me know of any I missed:

Voice Lists Average of Tau Average of Tau+
Mound Talk Community 1 0.364 107
The Cardinal Nation 1 0.340 196
Mound Talk 2 0.328 113
Fantasy Baseball Café 1 0.325 135
Baseball Notebook Fantasy 1 0.316 131
Minor League Ball Community 1 0.281 116
MLB 1 0.272 112
Baseball Digest Daily 1 0.270 112
Adam Foster 4 0.264 110
J.P. Schwartz 4 0.243 96
Jason Collette 2 0.234 118
Dayn Perry 4 0.233 127
Keith Law 3 0.229 89
David Regan 2 0.222 132
Mike Bornhorst 1 0.220 133
Jonathan Mayo 7 0.213 96
Kevin Goldstein 5 0.212 96
On Deck 3 0.199 119
John Sickels 4 0.193 131
Inside the Dugout 1 0.192 116
Prospects, P, and Suspects 1 0.188 224
SportsBlurb 1 0.187 109
Matt Garrioch 2 0.187 92
Lindy's 1 0.184 136
Dobber Baseball 1 0.180 104
Matthew Pouliot 6 0.174 104
Matt Hagen 2 0.168 82
Baseball Intellect 1 0.164 95
Aaron Gleeman 6 0.164 102
Baseball America 21 0.162 116
Warm October Nights 1 0.152 92
CBS 5 0.152 83
Frankie Piliere 1 0.148 86
Bob Reed 2 0.146 86
Mike Gullo 4 0.143 102
Sporting News 1 0.142 85
Baseball Notebook 2 0.140 71
Rotowire 2 0.139 106
Marc Hulet 1 0.132 76
Bryan Smith 4 0.127 66
Kevin Wheeler 1 0.124 110
Rany Jazayerli 8 0.124 117
Ken Warren 3 0.121 97
Sports Weekly 1 0.116 48
Spring Training 15 0.113 88
Team One 1 0.110 81
Diamond Futures 4 0.098 57
David Srinivasan 1 0.093 82
The Sporting News 2 0.092 63
David Cameron 2 0.086 95
Ken Phelps 3 0.082 71
Prospect Report 1 0.080 71
Prospect Watch 1 0.069 91
Shane Mills 1 0.062 73
Top Prospect Alert 3 0.060 60
Preview Sports 1 0.045 -177
Ultimate 1 0.011 43
MLB Prospects 1 0.010 13
Creative Sports 1 0.009 -8
Fastball 1 0.009 10

Project Prospect ranks the highest for absolute tau, but their lists cover a very good range of years for lists in general (2007-2010). When taking year into account, John Sickels (2000-2002) and Dayn Perry (2004-2007) look much better.

Conclusion

Have you ever read over an internet post

And the writer ain't no good?

I mean the grammar is sloppy, the words are mingled

And he can't be understood

-Wonder Mike Trout

The main purpose of this post is to get this research out there to begin a discussion about the best way to rank prospect lists. So how can we improve this? Is Kendall's Tau the best method to use? What is the best way to put all lists on equal footing? Should we account for the year? What about the length of the list? Let me know your ideas.

As it stands now, of lists ranked at least three times from 2007 to 2010-the most recent lists in this analysis- Jonathan Mayo (120) and Project Prospect (110) are the only lists to maintain an above-average rating. Top Prospect Alert comes in at 96 and Baseball America sits at 95, while Baseball Prospectus is at 92. ESPN brings up the rear at 89.

. . .

Statistics courtesy of Fangraphs and Baseball Prospectus. Thanks to Stuart Wallace for his help with R and Kendall's Tau.

Chris St. John is a writer at Beyond The Box Score. You can follow him on Twitter at @stealofhome.