Prospect's delight: Which top 100 list is best?

Introduction

Now what you read is not a test, I'm ranking prospect lists

And me, Rstats, and some math are gonna try to lift the mist

-Wonder Mike Trout

There has been some great analysis on how well Baseball America's prospect rankings translate to future success, see my Prospect Analysis page for examples from various writers around the web. Adam Foster at Project Prospect has done some work with this in his industry comparisons. However, BA isn't the only list around. Throughout the years, various rankings have come and gone. Out of all of these, which was the best prospect list?

Method

Check it out, it's the k-e-n-d-the-a-l-l

And the rest is t-a-u

You see, this is the code for the method of the post

And these reasons I'll bring to you

-Big Bank Hammerin' Hank

First, I compiled 166 prospect lists in the era from 1990 to 2010. I used 2010 as a cutoff to allow players four years to blossom in the major leagues. I found as many as I possibly could, knowing I missed many as well*. Once everything was compiled, the question becomes: how do you figure out which one is best? In my initial research into this, I found the book "Who's #1?" which suggests using Kendall's Tau or Spearman's weighted footrule when comparing ranked lists.

*If you have a list that you would like to be included in this, please e-mail me or contact me on Twitter @stealofhome.

Recently, Neil Paine used Spearman's rho on organizational prospect rankings at 538. However, I believe the fact that I have repeated rankings (I gave a rank of 101 to all prospects from the real list that did not appear on the historical lists) disallows me from using this.

I asked Tango the best way to do this and his idea was to create a fantasy draft for each year and allow each list to draft their first available player, then repeat this for each year and each possible order of drafting, with the list that drafted the highest average fWAR as the winner. This is probably the correct way to do this. However, if I waited until I had working code for this scenario, this research would not be completed until prospect lists (as well as me) were but a dim memory. However, if you are capable of putting that code together, I would be very interested in working with you to publish that research.

I landed on using Kendall's Tau for this analysis and will do my best to attempt to describe it. Note: I am not a professionally-trained statistician, so if you are and see any errors I make in my description, please let me know.

Kendall's tau shows how close two columns of rankings are to each other. It gives a range from -1 (meaning they are ranked in the opposite order) to 1 (they are the same). It compares the number of rankings below and above the rankings and takes into account ties. Here is a youtube video describing part of what it does, although this does not look at ties.

Since Kendall's Tau works by comparing one list to another, I had to find the "true" top 100 for each year. I did this by finding all players eligible for prospect lists each year and ranking them by their career fWAR. Then I compared that true top 100 list to the various prospect lists. This is an area that may need cleaning, as I don't have the proper data to do this perfectly.

If the player did not appear on the list in question, they received a ranking of 101. If the list went over 100 players, I only used their first 100 rankings. If a list ranked fewer than 100 players, I compared it with the true list of that length for that year (e.g., MLB's top 50 lists are only compared to the true top 50 prospects eligible that year). One more wrinkle in this analysis is that not all lists claim the same eligibility standards. Most rank anyone who would be a rookie, but there are some variations on that. This may hurt some lists in these places.

This is an interesting exercise in itself to see which prospects were missed and which players were lurking in the lower levels before exploding on the scene. For instance, Pedro Martinez had the best career of anyone eligible for prospect lists in 1991, but was nowhere to be found on BA's list. However, in 1992, he jumped into the top 10 with a strong performance in three levels of the minor leagues the previous year.

Name	True	BA
Pedro Martinez	1	101
Chipper Jones	2	49
Mike Mussina	3	19
Jeff Bagwell	4	32
Ivan Rodriguez	5	7
Andy Pettitte	6	101
Jim Thome	7	93
Jim Edmonds	9	101
Mike Piazza	10	101
Kenny Lofton	11	75
Jeff Kent	12	101
Luis Gonzalez	13	101
Brian Giles	14	101
Moises Alou	15	101
Jorge Posada	16	101
Bernie Williams	17	11
Carlos Delgado	18	101
Mariano Rivera	19	101
Chuck Knoblauch	20	72

BA did not have 11 of the top 20 eligible prospects in their 1991 list. However, many of these players were young at the time and did eventually appear on a BA top 100. Bernie Williams was ranked higher (11) than his true score (17). Jim Thome was ranked lower (93) than his true score (7). Kendall's Tau takes all of this into account and spits out a rank correlation of 0.157 - about average for the results I found.

Results

Well it's on and on and on on and on

This post don't stop until I make you yawn

-Master Dillon Gee

Before I get into the results, I want to add a personal note: Please do not use these rankings to trash writers who may rate poorly here. First, I believe this method can be improved upon, which may change the results a bit. Second, and most importantly, prospect analysis is really hard. These lists require hours of research and dedication, making phone calls and meeting scouts, poring over numbers, and doing whatever else it takes to rank players. These are a labor of love for each individual and it takes a lot of guts to put your name next to something that is both outdated almost immediately—thanks to the dynamic nature of young players—and destined to mostly fail-thanks to how hard the majors are.

With that said, here are the raw results after running Kendall's Tau on each of the lists in my database:

Year	Length	List	Tau	Tau+
2008	100	Rotowire	0.372	109
2008	100	Mound Talk	0.367	107
2008	100	Mound Talk Community	0.364	107
2008	100	Top Prospect Alert	0.355	104
2008	100	Baseball America	0.352	103
2008	100	Baseball Prospectus	0.348	102
2010	40	The Cardinal Nation	0.340	196
2008	100	ESPN	0.331	97
2007	50	MLB	0.330	137
2007	100	Fantasy Baseball Café	0.325	135
2008	100	Project Prospect	0.320	94
2007	50	Baseball Notebook Fantasy	0.316	131
2010	50	MLB	0.303	175
2002	50	John Sickels	0.297	390
2007	100	Mound Talk	0.289	119
2007	100	Minor League Ball Community	0.281	116
2007	100	FOX	0.278	115
2007	100	MLB	0.272	112
2007	100	Baseball Digest Daily	0.270	112
2007	100	Rotoworld	0.268	111
2007	100	Project Prospect	0.264	109
1997	100	Baseball America	0.263	114
2008	50	MLB	0.262	77
2009	100	Project Prospect	0.261	115
2004	50	CBS	0.260	160
2007	100	CBS	0.257	106
2007	100	Top Prospect Alert	0.256	106
2007	100	Baseball Prospectus	0.255	106
2009	100	ESPN	0.255	112
2001	40	Baseball Prospectus	0.249	185
2007	100	RotoJunkie	0.242	100
2009	100	Top Prospect Alert	0.241	106
2007	100	Baseball America	0.240	99
2005	50	MLB	0.239	139
2005	100	Minors First	0.233	135
2005	100	FOX	0.232	135
2005	100	RotoAmerica	0.229	133
2009	100	MLB Prospect Guide	0.228	100
2006	100	RotoJunkie	0.225	136
2009	50	MLB	0.225	99
2006	100	Rotoworld	0.225	136
2006	100	On Deck	0.222	134
2006	100	SI	0.220	133
2004	100	Rotoworld	0.220	135
2007	75	SI	0.219	90
1991	100	Spring Training	0.218	116
2006	100	RotoAmerica	0.216	130
2009	100	The Hardball Times Fantasy	0.214	94
2004	100	FOX	0.214	131
2010	100	Project Prospect	0.212	122
2006	100	FOX	0.208	126
2009	100	Baseball Prospectus	0.207	91
2005	100	Baseball America	0.206	120
2006	100	Baseball America	0.202	122
2003	50	Baseball Think Factory	0.201	178
2005	100	On Deck	0.200	116
1998	100	Spring Training	0.200	115
1997	100	Spring Training	0.199	86
1995	100	Baseball America	0.199	131
2006	100	FOX Fantasy	0.198	120
1993	100	Baseball America	0.198	134
2004	100	Spring Training	0.195	120
1999	40	Baseball Prospectus	0.194	230
2006	75	Inside the Dugout	0.192	116
2009	100	Baseball America	0.190	83
1999	50	Prospects, P, and Suspects	0.188	224
2004	100	Minors First	0.188	116
2005	100	SportsBlurb	0.187	109
2004	100	Ken Warren	0.185	114
2001	50	Lindy's	0.184	136
2004	100	Baseball America	0.183	113
2010	100	Dobber Baseball	0.180	104
2005	100	Rotoworld	0.179	104
2003	40	Baseball Prospectus	0.175	155
2004	100	On Deck	0.174	107
2007	50	Aaron Gleeman	0.173	71
2005	50	The Hardball Times	0.169	98
2004	100	The Sporting News	0.167	103
2010	100	Baseball America	0.166	96
2010	100	Baseball Intellect	0.164	95
1996	100	Baseball America	0.164	104
1991	100	Baseball America	0.157	84
2006	60	Bob Reed	0.156	94
1996	100	Spring Training	0.152	96
2006	100	Warm October Nights	0.152	92
2003	100	Rotowire	0.150	132
2010	100	AOL	0.148	86
1998	100	Baseball America	0.147	85
2010	100	MLB Prospect Guide	0.146	84
2006	50	The Hardball Times	0.146	88
2005	100	Diamond Futures	0.143	83
2006	50	Sporting News	0.142	85
2001	100	Baseball America	0.141	105
2007	100	Baseball Notebook	0.141	58
2006	100	Baseball Notebook	0.138	83
2005	60	Bob Reed	0.135	79
2006	75	Baseball Analysts	0.135	82
2003	100	Ken Warren	0.134	118
2003	100	Rotoworld	0.133	118
2010	100	Fangraphs	0.132	76
2003	100	Top Prospect Report	0.129	114
2005	100	CBS	0.129	75
2004	50	Rotowire	0.128	79
2003	100	The Sporting News Wheeler	0.124	110
2001	100	Ken Phelps	0.124	92
2003	100	Baseball America	0.123	108
2010	100	The Hardball Times Fantasy	0.122	71
1994	100	Baseball America	0.120	111
2010	100	Top Prospect Alert	0.119	69
2010	100	Baseball Prospectus	0.118	68
2007	100	Sports Weekly	0.116	48
2002	100	Baseball America	0.115	152
2004	100	Diamond Futures	0.111	68
2001	100	Team One	0.110	81
2004	90	Baseball Analysts	0.109	67
2001	100	CBS	0.109	81
2004	50	Baseball Prospectus	0.105	65
2004	50	MLB	0.105	65
1995	100	Spring Training	0.104	69
2001	100	Spring Training	0.104	77
2003	100	Minors First	0.102	90
2001	100	Top Prospect Alert	0.102	75
2010	100	ESPN	0.102	59
2006	100	Diamond Futures	0.101	61
1993	100	Spring Training	0.097	66
1994	100	Spring Training	0.097	89
2003	100	Ken Phelps	0.095	84
2004	50	The Hardball Times	0.094	58
2003	100	The Sporting News	0.093	82
2001	50	John Sickels	0.093	69
2002	100	Strike Three	0.087	114
2002	100	Spring Training	0.086	114
2002	40	Baseball Prospectus	0.086	113
2003	100	Strike Three	0.085	75
2005	50	Baseball Prospectus	0.083	49
2003	100	Prospect Report	0.080	71
2002	100	Top Prospect Alert	0.079	104
2006	50	Baseball Prospectus	0.075	45
1992	100	Spring Training	0.075	105
1999	100	Baseball America	0.074	88
2002	100	Prospect Watch	0.069	91
1992	100	Baseball America	0.067	95
2003	100	Spring Training	0.065	57
2000	100	Baseball America	0.064	255
1999	100	Shane Mills	0.062	73
1999	100	Spring Training	0.059	70
2002	100	Minors First	0.051	67
2002	100	Ken Warren	0.045	59
2000	100	Preview Sports	0.045	-177
2005	75	Baseball Analysts	0.043	25
2007	100	Diamond Futures	0.039	16
1990	100	Baseball America	0.038	139
2006	50	MLB	0.029	-18
2002	100	Ken Phelps	0.028	37
2000	40	Baseball Prospectus	0.024	94
2000	100	Spring Training	0.021	84
2002	100	The Sporting News	0.018	24
2002	100	Rotoworld	0.017	22
1990	100	Spring Training	0.016	61
2000	50	John Sickels	0.011	-45
2000	50	Ultimate	0.011	43
2002	100	MLB Prospects	0.010	13
2003	100	Creative Sports	0.009	-8
1999	100	Fastball	0.009	10
1999	100	CBS	0.004	-4
2000	80	Top Prospect Alert	0.000	2

Congratulations, John Sickels, your 2008 Rotowire list was the best of all time!

The first thing that jumps out from these results is that tau is very highly correlated to the year the list was posted. Since most lists are created from a very similar group of players each year, that is expected. What this really shows, then, is that 2007 and 2008 were good years for prospect lists in general. Because of that, I have also included a corrected tau, which accounts for the yearly average. This is tau divided by yearly tau times 100. This makes... John Sickels' 2002 Top 50 the best list.

This is what the average tau looks like by year:

There is a cyclical trend here, where the average tau generally increases from 1990 to 1997, decreases to 2000, increases to 2008, and has decreased since then. This has to do with many of the top players graduating into the higher levels of the minors and gaining more national attention. This then creates better lists (e.g., Pedro Martinez going from unranked to the top 10 in 1991).

Finally, this table summarizes these results by list "voice." I have done my best to match these up, but let me know of any I missed:

Voice	Lists	Average of Tau	Average of Tau+
Mound Talk Community	1	0.364	107
The Cardinal Nation	1	0.340	196
Mound Talk	2	0.328	113
Fantasy Baseball Café	1	0.325	135
Baseball Notebook Fantasy	1	0.316	131
Minor League Ball Community	1	0.281	116
MLB	1	0.272	112
Baseball Digest Daily	1	0.270	112
Adam Foster	4	0.264	110
J.P. Schwartz	4	0.243	96
Jason Collette	2	0.234	118
Dayn Perry	4	0.233	127
Keith Law	3	0.229	89
David Regan	2	0.222	132
Mike Bornhorst	1	0.220	133
Jonathan Mayo	7	0.213	96
Kevin Goldstein	5	0.212	96
On Deck	3	0.199	119
John Sickels	4	0.193	131
Inside the Dugout	1	0.192	116
Prospects, P, and Suspects	1	0.188	224
SportsBlurb	1	0.187	109
Matt Garrioch	2	0.187	92
Lindy's	1	0.184	136
Dobber Baseball	1	0.180	104
Matthew Pouliot	6	0.174	104
Matt Hagen	2	0.168	82
Baseball Intellect	1	0.164	95
Aaron Gleeman	6	0.164	102
Baseball America	21	0.162	116
Warm October Nights	1	0.152	92
CBS	5	0.152	83
Frankie Piliere	1	0.148	86
Bob Reed	2	0.146	86
Mike Gullo	4	0.143	102
Sporting News	1	0.142	85
Baseball Notebook	2	0.140	71
Rotowire	2	0.139	106
Marc Hulet	1	0.132	76
Bryan Smith	4	0.127	66
Kevin Wheeler	1	0.124	110
Rany Jazayerli	8	0.124	117
Ken Warren	3	0.121	97
Sports Weekly	1	0.116	48
Spring Training	15	0.113	88
Team One	1	0.110	81
Diamond Futures	4	0.098	57
David Srinivasan	1	0.093	82
The Sporting News	2	0.092	63
David Cameron	2	0.086	95
Ken Phelps	3	0.082	71
Prospect Report	1	0.080	71
Prospect Watch	1	0.069	91
Shane Mills	1	0.062	73
Top Prospect Alert	3	0.060	60
Preview Sports	1	0.045	-177
Ultimate	1	0.011	43
MLB Prospects	1	0.010	13
Creative Sports	1	0.009	-8
Fastball	1	0.009	10

Project Prospect ranks the highest for absolute tau, but their lists cover a very good range of years for lists in general (2007-2010). When taking year into account, John Sickels (2000-2002) and Dayn Perry (2004-2007) look much better.

Conclusion

Have you ever read over an internet post

And the writer ain't no good?

I mean the grammar is sloppy, the words are mingled

And he can't be understood

-Wonder Mike Trout

The main purpose of this post is to get this research out there to begin a discussion about the best way to rank prospect lists. So how can we improve this? Is Kendall's Tau the best method to use? What is the best way to put all lists on equal footing? Should we account for the year? What about the length of the list? Let me know your ideas.

As it stands now, of lists ranked at least three times from 2007 to 2010-the most recent lists in this analysis- Jonathan Mayo (120) and Project Prospect (110) are the only lists to maintain an above-average rating. Top Prospect Alert comes in at 96 and Baseball America sits at 95, while Baseball Prospectus is at 92. ESPN brings up the rear at 89.

. . .

Statistics courtesy of Fangraphs and Baseball Prospectus. Thanks to Stuart Wallace for his help with R and Kendall's Tau.

Chris St. John is a writer at Beyond The Box Score. You can follow him on Twitter at @stealofhome.