Nothing embodies uncertainty better than the NFL draft – teams generally have an idea of who will do well, but are never sure. The same is true of the third-party draft analyst industry, where a dozen draft experts put out big boards and team draft grades every year.
There’s a lot of value in that, but just as teams are held accountable for how they value players in the draft, we can hold analysts accountable to the way they value and discuss players.
People often argue that one can’t “judge a draft until three years out,” and if that’s the case, we can evaluate analyst big boards.
Last year, we looked at who constructed the best big boards for the 2014 draft. We’ll use the same methodology this year to evaluate the boards from the 2015 draft. The same caveats apply: one draft doesn’t tell us everything about an evaluator’s abilities, and big injuries to specific players can tank an evaluator’s rankings through no fault of their own.
With another year of data, that seems particularly true – no one did as well in the 2015 draft as they did in the 2014 draft — and some of the best predictors of the 2014 draft ended up doing poorly at ranking players the following year. The year-to-year correlation for performance in the 2014 draft and performance in the 2015 draft was negative.
This says quite a bit about what year-to-year variance can mean; evaluators that ranked Kevin White or Randy Gregory high were punished for White’s persistent injuries or Gregory’s off-field problems.
Last year, The Huddle Report produced the best board in 2014. For 2015, The Huddle Report did a respectable job but lost out to NFL Rough Draft. That fits, as they also matched the highest number of top 100 players on their big board to the NFL’s top 100 picks, as scored by The Huddle Report’s Top 100 contest.
The evaluators who are most impressive are those that beat the NFL draft itself — not necessarily because NFL teams are staffed with better evaluators, but because they have access to more information — including injury and psychological data — and control over how a player is used; if a player only fits a zone-style running scheme or is a two-gap specialist, they will likely fall below a player of equal skill who has more versatility if the evaluator is a third party.
The NFL doesn’t have to worry about that because they can draft a player high if they are perfect fits for their system and can realize that value without problems.
In 2015, a shocking 11 big boards beat the NFL draft. The year prior, only five boards — as well as the consensus boards — did. Interestingly, no evaluator beat the NFL draft in both 2014 and 2015. Full results for the 2015 draft below:
The Consensus Big Board, the Evaluator Board and the Forecaster Board —
all explained here — that we’ve developed have consistently done well, predicting performance about as well as the NFL draft. In 2014, they beat the NFL draft in predictive performance.
Many of the players that tripped analysts up the most in the 2015 draft are predictable. No one was bold enough to argue that running back David Johnson should have been worth a first-round pick, but the value he’s demonstrated thus far shows as much. Eric Galko at Optimum Scouting and James Christensen with NEPatriotsDraft were the closest, however, with both putting him in their top 50.
Grady Jarrett also outperformed expectations, and he’s been playing as one of the best picks of the draft. Josh Norris of Rotoworld was the only one to slot him in his top 32 and he earns marks for recognizing that talent.
On the other hand, White and Paul Dawson hurt evaluators. No evaluator ranked White all that low, so no one lost out compared to the rest of their analysts, but Dawson let a few analysts down while Corey Chavous, Gil Brandt, Dan Shonka and Patrick Conn all did the best job avoiding the potential pitfall — none of them had Dawson in their top 100, and Chavous ranked him 183rd.
On the other hand, evaluators absolutely nailed Leonard Williams, who has been playing at an incredibly high level for the New York Jets. Evaluators were largely correct with Williams at No. 1, Brandon Scherff inside the top 10 and Melvin Gordon as a mid-first round talent.
Because of the high value of quarterbacks, the evaluators who had Jameis Winston and Marcus Mariota as high-level talents all benefited, though not many kept either of those two players outside of the top ten.
Outside of the first round, evaluators should generally be happy with how they rated Jordan Phillips, Laken Tomlinson, Nate Orchard and Tevin Coleman.
We’ve also learned that polarizing players are polarizing for a reason — they truly seem to boom or bust. There wasn’t much in terms of middle-of-the-road performance for polarizing, high-ranked players like Todd Gurley (boom), Danny Shelton (boom), Brandon Scherff (boom), Dante Fowler (bust), Shane Ray (bust), Randy Gregory (bust) and Dorial Green-Beckham (bust).
Though some players who drew opposite takes from analysts have split the difference — players like Byron Jones and Mariota may not have met the full extent of their draft hype, but they’ve hit their consensus rank; Mariota ranks as roughly the fifth-best player from the draft and ranked third on the consensus board. Jones has ranked as the 41st player on the consensus board and ranks 40th overall in value from the draft calls.
For the most part, however, a list of the most polarizing players from the draft results in big hits or big misses.
Interestingly, the two least polarizing first-round talents on the consensus board — Eli Harold and White — have both failed to meet their draft rankings fairly spectacularly (though Harold, ranked No. 32 on the consensus board, was largely correctly pegged by the NFL draft when picked 79th overall).
What’s even more fascinating than the results of the 2015 draft is our ability to combine performances of analysts across both the 2014 and 2015 drafts:
Once again, the combined draft boards we’ve developed outperform the pack and do almost identically as well as the NFL draft itself. Other than that, NFL Rough Draft, the Huddle Report and NFL Draft Geek have all outperformed the NFL draft itself.
Those evaluators deserve praise, but those at the middle or bottom of the list shouldn’t feel too downtrodden. The evidence from two different drafts suggests that we don’t know the true strengths or weaknesses of an evaluator until they’ve had a couple of drafts under their belt — four or five, in all likelihood.
For us, that means we should take in as much as we can from as many draft analysts as possible, and for NFL teams probably means that they should wait a little while before letting go of a general manager.
https://zonecoverage.com/2018/featur...015-nfl-draft/