Draft "experts" , Do they know a damn thing?

[hometeam]

http://harvardsportsanalysis.wordpre...raft-rankings/

Equally Inaccurate: An Analysis of Mel Kiper Jr. and Todd McShay’s Draft Rankings
Posted on May 25, 2012 by harvardsportsanalysis
By Kevin Meers and Scott Sherman

Conan vs. Leno. Jack Shephard vs. John Locke. Seinfeld vs. Newman. Over the years, hundreds of great rivalries have come to dominate our television screens. But to sports fans, none of those individual competitions are quite as captivating as the one between ESPN’s two draft gurus: Mel Kiper Jr. and Todd McShay.

There are many iconic hairstyles in the National Football League. Brett Keisel’s lumberjack beard. Troy Polamalu’s bodacious curls. Clay Matthews’ scintillating golden locks. But among the most distinct looks is that of Kiper, whose hair, glasses, and those squinted eyes are one of the signature images of the NFL draft. Every year, he and his fellow heavyweight, the former University of Richmond waterboy McShay, slug it out on millions of television screens in the months leading up to the draft a dramatic attempt to display their expertise. But which of these two titans is a better judge of talent? This post sought to answer that question.


To compare the success rates of the two gurus, we collected the rankings of the Top 25 players in each year’s draft from 2006 (when McShay joined ESPN) through 2010. Kiper’s rankings were determined by his final “Big Board” before each year’s draft, while McShay’s were determined via Scouts Inc.’s final Top 25 (McShay runs Scouts Inc.). We then compared each player’s ranking to their actual performance ranking in their draft class (performance here is approximated by Career Approximate Value). As an example, Kiper ranked Julius Peppers as the best player in the 2002 NFL Draft, and that prediction proved to be dead on, as 10 years later Peppers has the highest CAV of any player from the ’02 draft. While we would have preferred to use the ranks of all players drafted, Kiper only releases his top 25 prospects on his “Big Board,” which severely limited sample size. Nonetheless, here’s what we found:




As the chart above shows, Kiper and McShay have proven to be equally inefficient judges of talent over the past five years. Neither expert’s rankings were a particularly good predictor of how a player would perform compared to the rest of his class. In fact, the average errors listed above suggest that the gurus’ rankings are off by around 35 spots when compared to the player’s actual performance. In other words, a player Kiper or McShay ranked as the 15th-best player in the draft is most likely to actually have been the 40th-best based on CAV to this point in their careers. The root mean square errors (RMSE) imply that Kiper is very slightly less prone to extreme errors than his ESPN counterpart, but that difference is not significant. RMSE severely punishes large errors, so having a comparatively lower RMSE implies that one makes fewer huge errors. However, like the average errors, the RMSE for each scout is enormous.

Indeed, when looking at the lists themselves, it is clear that both Kiper and McShay have, like all draft experts, been prone to extreme hits and misses when compared to the other. In 2006, both had future busts Matt Leinart, A.J. Hawk, Vince Young, and Michael Huff ranked in their Top 10s. But McShay was wise enough to include future All-Pro center Nick Mangold in his Top 25, while Kiper instead opted for Jason Allen. The following year, McShay found more success by ranking Darrelle Revis tenth overall, but the Island himself was nowhere to be found on Kiper’s Big Board. Also that year, both Kiper and McShay smartly pegged Calvin Johnson, Adrian Peterson, and Joe Thomas as three of the four top players in the draft, but both also had JaMarcus Russell and Brady Quinn in their top sevens.

In 2008, Kiper wisely ranked future bust Derrick Harvey 23rd—McShay had him 11th—but Kiper had fellow pass-rushing disaster Vernon Gholston 7th, compared to 15th for McShay. In 2009, both had the bust triumvirate of Aaron Curry, Michael Crabtree, and Jason Smith in their top four, but Kiper pegged Matthew Stafford at No. 3 while McShay had him at No. 7. McShay, however, wisely had both Clay Matthews and Hakeem Nicks in his Top 25, while Kiper had neither (but did, unlike McShay, include Percy Harvin). Finally, in 2010, Kiper, unlike McShay, smartly deemed Ndamukong Suh a better prospect than Gerald McCoy, but that success was immediately negated by his ranking of Jimmy Clausen at No. 4 overall (McShay, to his credit, did not feel Clausen was a top 25 talent).

Clearly, then, both “experts” have been both on-point and wildly off-base with their draft predictions over the past few years. On the whole, neither’s rankings are very accurate. If a player is listed in the Top 25 by either scout, that players will, on average, be in the top forty or sixty players from their draft class, but he is absolutely not guaranteed to be one of the very best players from his year. Furthermore, none of the differences between the average errors of Kiper and McShay are statistically significant: the differences between them are likely more due to luck than actual skill. In other words, you could take Kiper’s “Big Board” or McShay’s Top 25 and ask any random person to put them in any random order, and on average, that person’s rankings will be about as accurate as predicting future NFL success than any of ESPN’s two experts.

Of course, this study was limited by its sample size, as we’d have a fair better chance of really determining who was more accurate if Kiper and McShay both ranked every player in the draft (which only Scouts Inc. does; Kiper merely publishes his Top 25). But the nearly-equal numbers in the table above suggest there is not much of a difference between either’s ability to form an accurate Top 25. Of course, predicting NFL success—as it is in any sport—is extremely difficult, and it’s certainly possible that Kiper and McShay are statistically more accurate than any of the dozens of other websites that publish draft rankings each year. Since player rankings tend to be generally similar, it’s likely all such sites would be prone to a number of mistakes. But to determine whether such errors were as significant as those made by Kiper and McShay, future research should compare ESPN’s two gurus success rate to that of their lesser-known counterparts around the web. Until then, the best we can say is that neither Kiper or McShay is extraordinarily good at what they are paid to do, or even better than the other at doing it.

This article specifically targets McShay and Kiper, but it got me thinking about whether any so called draft experts knew shit about anything. Do they know more than me and you?

Are there any other examples or data on whether any draft expert or non-expert actually gets it right consistently?

So picking a QB who is 'ranked' at X spot in the first round, but not 1st if he is the best guy out there is stupid based on what? Mel Kipers big board? Get real. Nobody knows shit!










Geno for president

[Cornstock]

Picking the top 25 in order of who will turn out the best is like picking a perfect 65 team bracket in NCAA basketball. Except you don't know exactly which "teams" are in the "tourney".

[KCtotheSB]

I prefer we just pick a name out of a hat.

[HemiEd]

Its a process.

[Chiefnj2]

The problem is partially that their final boards and rankings are heavily influenced by their sources who tell them which team is drafting and targeting certain players. This way on draft day when their top 25 are all gone by the 30th pick in the draft - they look smart to the average viewer.

[htismaqe]

Originally Posted by Chiefnj2

The problem is partially that their final boards and rankings are heavily influenced by their sources who tell them which team is drafting and targeting certain players. This way on draft day when their top 25 are all gone by the 30th pick in the draft - they look smart to the average viewer.

Yeah, that's why you have to track the changes between now and the draft.

Guys like Rick Gosselin have very little variance.

Guys like Kiper make WHOLESALE changes to their boards.

[Chiefnj2]

Originally Posted by htismaqe

Yeah, that's why you have to track the changes between now and the draft.

Guys like Rick Gosselin have very little variance.

Guys like Kiper make WHOLESALE changes to their boards.

Does Gosselin even have a board now?

Are you aware of any "expert" who has a clue with regard to QB's? And lets face it, part of the problem is the team that drafts a player. Wilson wouldn't have gotten the opportunity to win the starting job in KC. He'd still be viewed as a wasted reach if KC drafted him.

[htismaqe]

Originally Posted by Chiefnj2

Does Gosselin even have a board now?

Are you aware of any "expert" who has a clue with regard to QB's?

No, because Gosselin ONLY listens to his sources. He doesn't try to be scout, he's in the prediction business.

What gets guys like Kiper in trouble is that they try to pretend to be scouts and then when it gets close, they change it all up. Kiper wants people to believe he's a scout when in reality, he too is in the prediction business.

I think Scott Wright does a fairly good job of analyzing players, not just QBs. He's had Barkley and Smith 1/2 for most of the season.

[WhiteWhale]

Draft 'experts' are just as likely to know who good prospects are as fans are.

I'm not convinced teams are much better either.

That's why the draft is fun for people who really like college ball. The annoying ones are the guys who read the primer and think they're an expert. I don't do either as I'm no draftnik, but I certainly know some fans are a lot better at this than others....

[Chiefnj2]

Originally Posted by htismaqe

No, because Gosselin ONLY listens to his sources. He doesn't try to be scout, he's in the prediction business.

What gets guys like Kiper in trouble is that they try to pretend to be scouts and then when it gets close, they change it all up. Kiper wants people to believe he's a scout when in reality, he too is in the prediction business.

I think Scott Wright does a fairly good job of analyzing players, not just QBs. He's had Barkley and Smith 1/2 for most of the season.

The nice thing about Wright is he keeps his archives accessible, but he's as bad with QB's as anyone. His top two QBs each year with their overall ranking:

2007
Quinn #2 prospect (miss)
Russell #5 (miss)

2008
Ryan #6 (hit)
Brohm #28 (miss, shouldn't have been in top 100)

2009
Stafford #1 (I'll say hit, some will say no)
Sanchez #6 (miss)

2010
Clausen #10 (miss)
Bradford #11 (Incomplete maybe?)

2011
Gabbert #9 (miss)
Locker #26 (too early?)
Newton #29 (miss, too far back. Should be ahead of 1 and 2)

2012
He, like the rest of the world, liked Luck (1) and Griffin (5) but had Wilson at #152.

[Dave Lane]

I think they are like every team out there. 30% success is about it. Some years its 0% some years you hit 60% but combined even the hive mind of CP its pretty much who knows.

[duncan_idaho]

Anybody check out James Christensen's stuff?

He runs this site: Patriots Draft

Seems to be pretty legit, and I like the way they list their draft profiles, especially the QB stuff.

He also has Geno at the top of his board. Link to his scouting report on Geno: here.

[chop]

Originally Posted by duncan_idaho

Anybody check out James Christensen's stuff?

He runs this site: Patriots Draft

Seems to be pretty legit, and I like the way they list their draft profiles, especially the QB stuff.

He also has Geno at the top of his board. Link to his scouting report on Geno: here.

He has the the Chiefs taking Tyler Wilson over Geno Smith with the first overall pick.

[patteeu]

Originally Posted by duncan_idaho

Anybody check out James Christensen's stuff?

He runs this site: Patriots Draft

Seems to be pretty legit, and I like the way they list their draft profiles, especially the QB stuff.

He also has Geno at the top of his board. Link to his scouting report on Geno: here.

Interesting: He's got Sheldon Richardson going 3rd overall to the Raiders.

[patteeu]

Originally Posted by chop

He has the the Chiefs taking Tyler Wilson over Geno Smith with the first overall pick.

I don't see that. He's got Geno Smith going to the Chiefs if Brandon Albert is retained. But he says "Luke Joeckel has to be the pick" if Albert leaves.

Edit: It looks like that site has 4 different guys doing mocks. The one duncan_idaho mentioned has the Chiefs taking Geno. So do two of the others. The fourth guy has the Chiefs taking Matt Barkley.

[duncan_idaho]

Yeah, its several guys in collaboration. Their profiles are legit. And the scouting reports certainly track well (I used to edit Russ Lande, and I think this stuff is just as good/easier to read).