Graph of the Day: Comparing Power Rankings

by Sky Kalkman on Jun 24, 2009 2:00 PM EDT

Power Rankings are funny. They all try to measure something different and cater to a different audience. And no matter the system, fans of some team will disagree vehemently. Nonetheless, i was curious to see how Justin's BtB rankings compared to other systems.

The comparison set: Pythagorean wins, second- and third-order wins from BPro, BPro's Hit List factor (which is simply an average of actual record, Pythag, and second- and third-order winning percentage), and RPI. BtB_lg removes the league adjustment.

All systems are compared to actual records as of late last week using both correlation and total absolute error. Now, "error" isn't a bad thing here, because these power rankings aren't trying to predict past record, they're trying to extract true talent and ignore fluke.

In general, as correlation goes up, aboslute error should go down. One system where that doesn't happen is RPI, which correlates the closest to actual record, but has a much tighter set of ratings, making its absolute error quite high. The system that matched actual records the least was BtB's, followed by the non-league-adjusted BtB number and the two most adjusted BPro systems. Personally, I'd like to see a power ranking NOT match actual record that closely, because I want new information. Of course, not matching closely doesn't mean the system is any good.

Anyway, my major takeaways from this graph are that BtB's system definitely doesn't just match win-loss record (as many commenters have creatively pointed out) and that RPI and BPro's systems DO.

Tweet Comment 8 comments | 0 recs |

Do you like this story?

Comments

Sweet!

My system is the least close to reality of all the systems!!

The reason, of course, is source data. Other systems are using win or run totals, and we’re estimating those data from more fundamental data. I think that’s why I like this system. If your first look at team performance are true team records, and the second look is Pythagorean record, these rankings provide a third, novel look at the same teams.
-j

My blog: Basement-Dwellers.com

by JinAZ on Jun 24, 2009 4:42 PM EDT reply actions

I agree. You're not measuring wins. You're measuring the things that should combine into wins.

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jun 25, 2009 12:20 AM EDT up reply actions

First off, what is RPI?

Secondly, it doesn’t seem to make sense to compare it to actual records because those aren’t really what you want out of a power ranking. Justin’s seems to be the most thorough, I would say it’s probably a bit better than 3rd Order Wins because it values pitching using tRA instead of just reverse EqA.

Also, and I don’t know how easy this would be to do, but wouldn’t it be better to compare it to a full seasons record? You could use each formula and calculate final rankings using last years teams, then compare those to the final record.

St. Louis relievers... defying win expectancy since 2008
http://www.drivelinemechanics.com/

by vivaelpujols on Jun 24, 2009 6:40 PM EDT reply actions

I agree with all that.

I wasn’t comparing the rankings to actual record to decide how good they were. I was comparing them to see how much information other than actual wins they contained.

Beyond the Boxscore Not a member? Sign up.

by Sky Kalkman on Jun 25, 2009 12:21 AM EDT up reply actions