Last year, we looked at what the consensus of the scouting community had to say when it came to which players were the best prospects in the NFL draft. That look alone gave us a lot of tools, which meant not only could we have a unique way to “grade” drafts, take a look at which teams were off-the-wall, and which evaluators were best at “predicting” the NFL draft.

The Huddle Report has a similar competition, grading mock drafts as well as top 100s. Draft Board Guru won their competition—their top 100 had the most players actually selected in the top 100. We’ll take a look at other ways to grade that process in a little bit as well.

We’ll take a look at last year’s results, then follow it with a preview of this year’s board. If you want to skip to this year’s board, click here.


 

2014 Review

Awards

“Gold Standard”

In this case, the Gold Standard doesn’t refer to who best ranked the players in terms of how well they’ll do, but whose rankings best captured the feeling of the evaluation community. There are a number of divergence tests to run on this sort of thing, but instead of boring you with process I’ll just let you know that they all came to the same conclusion.

DraftTek’s final board was the best at capturing the thoughts of the evaluation community as a whole, which means that if you only had one board to look at to see what people think of your favorite prospect, you’d check out the one they had at DraftTek. This year, their board is here.

 

“Odd Duck”

A lot of people take pride in the fact that their evaluations are independent of others, and well they should. Many times, that produces unusual rankings, and gives their boards the best opportunity to surpass others when it’s finally graded—or fall well behind. If you want unique takes on the draft, you want the Odd Duck.

Last year’s Odd Duck didn’t have a close competitor. Kyle Crabbs at NDT Scouting produced the most divergent talent ranking for NFL prospects last year in his guide. The guide is available for purchase for $10.00. It’s thoughtful, well-organized and easy to read. Here’s a sample of 18 pages from this year’s guide, which includes a number of evaluations and an explanation of their methdology.

 

Best Prediction

The Huddle Report’s contest is very good, and a pretty handy reference for figuring out who the players in the top 100 will go. They look at the players in a board’s Top 100, then count up how many of those players actually went in the first 100 picks.

We’ll take it a step further, this time by figuring out how far away a player was actually picked from their spot on the board and adding up all those differences. We’re adjusting for how high a player is picked, so if a player ranked first goes fifth, it means a much bigger penalty than if a player ranked 75 goes 88th.

For this we’ll exclude those boards that had at least 100 players, something we’ll exclude in all phases this year, not just from awards, but input into the rankings. That means the winner, Daniel Jeremiah, doesn’t count because he only ranked 50 players.

The next best predictor was Mike Mayock at the NFL Network, whose score was actually fairly stunning. Despite the fact that he had Jadeveon Clowney as his second-ranked player (one of 11 rankers to do so out of 34), Greg Robinson as his third-ranked player and Blake Bortles as his 15th-ranked player, the rest of his rankings matched the board fairly well.

 

“Out of Sync”

Instead of calling the opposite award the “Worst Prediction,” we’ll call it Out of Sync, because most of these evaluators aren’t attempting to predict the Top 100 with their list of top 100 players, simply predict the best 100 players. Kyle Crabbs won that award as well, and got there with some bold moves—ranking third overall pick Blake Bortles as his 49th-best player, second-overall pick Greg Robinson as his tenth-ranked player and fourth-overall pick Sammy Watkins as his 16th-ranked player.

Oh, and Mike Evans ranked 123rd.

 

Hypothesis-Testing

Evalucasting?

Last year, we divided the boards into “forecasters” and “evaluators” in order to separate the two approaches that seemed to have developed when ranking draft prospects. The first set of boards come from what some people have started calling “Big Draft,” which is a reflection of dominant media narratives—we usually see them on TV on ESPN or the NFL Network, or online at those places in addition to CBS and on occasion Yahoo!

The idea was that those that are “plugged in” to the league are not necessarily better at evaluating players—although they could be, due to training, access (to coach’s film or scheme) and resources (like former players)—but have information most other sources wouldn’t have, like off-field concerns and injury. They know how a player did in interviews and have some sense of what teams are thinking.

That means their access to the pulse of the NFL will influence their evaluations either implictly or explicitly, and results in a board that is reflective of NFL opinion.

There are a couple of things we could do here. First, we could run the Top 100 test we did above to see if the forecaster (or evaluator) board beat Mike Mayock.

After that, we could take a look at where the evaluator boards diverged with the forecaster boards, and see who was closer to the actual pick. We’ll count up “wins” for those who were closer to the actual pick, and create “winning percentages” for the Top 32, Top 50, Top 100, Top 256.

So, was the forecaster board more accurate than Mayock? Yes, but by the tiniest margin. The average error for Mike Mayock was the magnitude of error that would have the player who was picked 20th was ranked 34th, which is actually pretty good on average.

The average error for the forecaster board was as if that 20th pick was ranked 33rd. It’s a very small difference. A better example comes at the 100th pick. The average error for Mayock would rank the player who ended up going 100 was ranked 141st. For the forecaster board, that player would be ranked 139th.

Either way, the forecaster board is pretty good at predicting the draft from that perspective.

At the end of the day, however, it’s not really what you’re looking for when figuring out where a player will be drafted. If a player is ranked 35th, 38th and 32nd by different groups, there’s not a real controversy. Instead, it’s when a player is highly lauded by one set of draftniks and derided by the other that it’s really interesting.

Yes, I’m talking about Teddy Bridgewater. Sort of.

So, we’ll create win counts for those bigger differences for each group of players: Top 32, Top 50, Top 100 and Top 256. As an example, Teddy Bridgewater was ranked third by the “evaluators” and 18th by the “forecasters.” Bridgewater was drafted 32nd, so the second set of boards “won.”

Evaluator Wins Forecaster Wins Ties
Top 32 0 2 1
Top 50 0 4 1
Top 100 1 9 1
Top 256 9 31 1

The difference is clear. The evaluators didn’t win once in the top 32 or the top 50, which is pretty compelling evidence that the forecasters do indeed forecast.

Evaluator Win% Forecaster Win %
Top 32 16.7 83.3
Top 50 10 90
Top 100 13.6 86.4
Top 256 23.2 76.8

It’s a landslide. The tie, by the way, was Taylor Lewan—who was ranked 15th by the evaluators and 7th by the forecasters, and ended up going 11th, splitting the difference. There’s a good case for eliminating ties from the win percentage calculation, but there’s not much point: the difference is clear.

 

But Are They Right

I think the best way to see how to evaluate which board best predicts player performance is to take a look a the players who they disagreed on and make a judgment call on that player’s performance vs. the value of the spot he was projected to be in.

Naturally, that would cause a lot of disagreement and is not foolproof. People disagree on what the value of a third-round pick is, and what teams should expect from them. Is a high-quality backup a good pick in the third? A bad starter? Sometimes those two aren’t distinct, but the second is judged more harshly. I would consider, for example, Mason Foster to be in that category.

Perhaps the best way to resolve the evaluative tension behind the value of some of these picks isn’t necessarily to use Approximate Value (my go-to way of answering these questions) but to index Approximate Value against Pro Football Focus scores to see what the expected PFF score for a pick should be. Then, we can use that as a rough guide to see who was closer on some of these controversial players.

That will have to wait, however, as these players get more time in the NFL. For now, let’s make snap judgments on the biggest disagreements.

Evaluator Forecaster Winner
Mike Evans 16 7 Forecaster
Justin Gilbert 26 9 Evaluator
Taylor Lewan 16 6 Forecaster
Calvin Pryor 32 17 Forecaster
Jason Verett 18 37 Evaluator
Teddy Bridgewater 3 18 Evaluator
Joel Bitonio 84 42 Forecaster
Derek Carr 12 31 Evaluator
Jeremy Hill 96 54 Forecaster

Well, alright then.


 

2015 Preview

The Big Board

Well if you’re here for the big board, here it is. It’s incomplete and needs about five more rankings groups to contribute in order to finish it. Tomorrow we’ll have it updated and complete, along with a complete analysis of the different ways to rank the boards (logarithmic point distributions, medians, means and trimmed means) and the differences between the evaluators and forecasters in this year’s draft.

Two notes before you see it. The first is that we have “roles” and “positions” to designate the likely fit for players to be draft and the likely positions they’ll play. Roles are a narrow grouping than positions, so all 3-technique tackles, 5-technique tackles and nose tackles are “IDL” or interior defensive linemen.

Sometimes, those roles and positions are the same, like for edge rushers and QBs. Here’s a cheatsheet:

Position-Role Cheat Sheet

I’ll explain the groupings and why they are the way they are tomorrow as well as the analysis of the rankings I mentioned earlier. Remember, Anthony Barr was an “edge” player last year, so the position groupings aren’t ironclad. Still, if you think a player’s role was misdefined, do not hesitate to let me know.

We’ll have a live, complete big board running as part of our draft coverage. It’s going to be a lot of fun.

Here’s the full board (missing a few entrants):

Rank Player School Role Role Rk Position Pos Rk
1 Leonard Williams USC 5T 1 IDL 1
2 Jameis Winston Florida St QB 1 QB 1
3 Marcus Mariota Oregon QB 2 QB 2
4 Amari Cooper Alabama FL 1 WR 1
5 Dante Fowler Jr. Florida EDGE 1 EDGE 1
6 Kevin White (WR) West Virginia SE 1 WR 2
7 Brandon Scherff Iowa OT 1 OL 1
8 Randy Gregory Nebraska EDGE 2 EDGE 2
9 Vic Beasley Clemson EDGE 3 EDGE 3
10 Todd Gurley Georgia RBF 1 RB 1
11 DeVante Parker Louisville SE 2 WR 3
12 Alvin Dupree Kentucky EDGE 4 EDGE 4
13 Trae Waynes Michigan St OCB 1 CB 1
14 Danny Shelton Washington NT 1 IDL 2
15 Shane Ray Missouri EDGE 5 EDGE 5
16 La’el Collins LSU OT 2 OL 2
17 Landon Collins Alabama SS 1 S 1
18 Marcus Peters Washington OCB 2 CB 2
19 Melvin Gordon Wisconsin RBF 2 RB 2
20 Malcom Brown Texas NT 2 IDL 3
21 Andrus Peat Stanford OT 3 OL 3
22 Ereck Flowers Miami (FL) OT 4 OL 4
23 Eddie Goldman Florida St NT 3 IDL 4
24 Arik Armstead Oregon 5T 2 IDL 5
25 Cameron Erving Florida St OC 1 OL 5
26 D.J. Humphries Florida OT 5 OL 6
27 Jaelen Strong Arizona St FL 2 WR 4
28 Eric Kendricks UCLA OFOB 1 OFB 1
29 Shaq Thompson Washington OFOB 2 OFB 2
30 Eli Harold Virginia EDGE 6 EDGE 6
31 Kevin Johnson Wake Forest OCB 3 CB 3
32 Jake Fisher Oregon OT 6 OL 7
33 T.J. Clemmings Pittsburgh OT 7 OL 8
34 Dorial Green-Beckham Missouri SE 3 WR 5
35 Jalen Collins LSU OCB 4 CB 4
36 Owamagbe Odighizuwa UCLA EDGE 7 EDGE 7
37 Maxx Williams Minnesota TE 1 TE 1
38 Jordan Phillips Oklahoma NT 4 IDL 6
39 Nelson Agholor USC FL 3 WR 6
40 Breshad Perriman Central Florida SE 4 WR 7
41 Michael Bennett (DT) Ohio St 3T 1 IDL 7
42 Preston Smith Mississippi St EDGE 8 EDGE 8
43 Byron Jones Connecticut OCB 5 CB 5
44 Benardrick McKinney Mississippi St OFIB 1 OFB 3
45 Carl Davis Iowa NT 5 IDL 8
46 P.J. Williams Florida St OCB 6 CB 6
47 Devin Funchess Michigan SE 5 WR 8
48 Cedric Ogbuehi Texas A&M OT 8 OL 9
49 Jay Ajayi Boise St RBF 3 RB 3
50 Devin Smith Ohio State SE 6 WR 9
51 Phillip Dorsett Miami (FL) SE 7 WR 10
52 Denzel Perryman Miami (FL) OFIB 2 OFB 4
53 Tevin Coleman Indiana RBF 4 RB 4
54 Ronald Darby Florida St OCB 7 CB 7
55 Duke Johnson Miami (FL) RBF 5 RB 5
56 Paul Dawson TCU OFIB 3 OFB 5
57 Nate Orchard Utah EDGE 9 EDGE 9
58 Danielle Hunter LSU EDGE 10 EDGE 10
59 Quinten Rollins Miami (OH) OCB 8 CB 8
60 Eric Rowe Utah OCB 9 CB 9
61 Ameer Abdullah Nebraska RBF 6 RB 6
62 A.J. Cann South Carolina OG 1 OL 10
63 Sammie Coates Auburn FL 4 WR 11
64 Brett Hundley UCLA QB 3 QB 3
65 Laken Tomlinson Duke OG 2 OL 11
66 Stephone Anthony Clemson OFIB 4 OFB 6
67 T.J. Yeldon Alabama RBF 7 RB 7
68 Alex Carter Stanford OCB 10 CB 10
69 Clive Walford Miami (FL) TE 2 TE 2
70 Mario Edwards Jr. Florida St EDGE 11 EDGE 11
71 Ifo Ekpre-Olomu Oregon NCB 1 CB 11
72 Hau’oli Kikaha Washington EDGE 12 EDGE 12
73 Grady Jarrett Clemson 3T 2 IDL 9
74 Tre’ Jackson Florida St OG 3 OL 12
75 Xavier Cooper Washington St 5T 3 IDL 10
76 Rashad Greene Florida St FL 5 WR 12
77 Tyler Lockett Kansas St FL 6 WR 13
78 Trey Flowers Arkansas EDGE 13 EDGE 13
79 Henry Anderson Stanford 5T 4 IDL 11
80 Ty Sambrailo Colorado St OT 9 OL 13
81 Donovan Smith Penn St OT 10 OL 14
82 Damarious Randall Arizona State FS 1 S 2
83 Ali Marpet Hobart OG 4 OL 15
84 David Johnson Northern Iowa RBC 1 RB 8
85 Justin Hardy East Carolina FL 7 WR 14
86 Lorenzo Mauldin Louisville EDGE 14 EDGE 14
87 Markus Golden Missouri EDGE 15 EDGE 15
88 Hroniss Grasu Oregon OC 2 OL 16
89 D’Joun Smith Florida Atlantic NCB 2 CB 12
90 David Cobb Minnesota RBF 8 RB 9
91 Derron Smith Fresno St FS 2 S 3
92 Senquez Golson Ole Miss NCB 3 CB 13
93 Jaquiski Tartt Samford SS 2 S 4
94 Daryl Williams Oklahoma OT 11 OL 17
95 Mike Davis South Carolina RBF 9 RB 10
96 Tre McBride William & Mary SE 8 WR 15
97 Josh Shaw USC OCB 11 CB 14
98 Doran Grant Ohio State OCB 12 CB 15
99 Jeff Heuerman Ohio St TE 3 TE 3
100 Bryce Petty Baylor QB 4 QB 4
101 Kwon Alexander LSU OFIB 5 OFB 7
102 Cody Prewitt Ole Miss FS 3 S 5
103 Jeremiah Poutasi Utah OG 5 OL 18
104 Za’Darius Smith Kentucky EDGE 16 EDGE 16
105 Garrett Grayson Colorado State QB 5 QB 5
106 Kevin White (CB) TCU NCB 4 CB 16
107 Gabe Wright Auburn 3T 3 IDL 12
108 Arie Kouandjio Alabama OG 6 OL 19
109 Jeremy Langford Michigan St RBF 10 RB 11
110 Jamison Crowder Duke SWR 1 WR 16
111 Anthony Harris Virginia SS 3 S 6
112 Marcus Hardison Arizona St 3T 4 IDL 13
113 Javorius Allen USC RBF 11 RB 12
114 Gerod Holliman Louisville FS 4 S 7
115 Rob Havenstein Wisconsin OT 12 OL 20
116 Tyler Kroft Rutgers TE 4 TE 4
117 James Sample Louisville SS 4 S 8
118 Charles Gaines Louisville NCB 5 CB 17
119 Anthony Chickillo Miami (FL) EDGE 17 EDGE 17
120 John Miller Louisville OG 7 OL 21
121 Jacoby Glenn Central Florida OCB 13 CB 18
122 Jordan Hicks Texas OFIB 6 OFB 8
123 Ty Montgomery Stanford SWR 2 WR 17
124 Reese Dismukes Auburn OC 3 OL 22
125 Chris Conley Georgia SE 9 WR 18
126 Andy Gallik Boston College OC 4 OL 23
127 Tony Lippett Michigan State FL 8 WR 19
128 Josue Matias Florida St OG 8 OL 24
129 Ben Heeney Kansas OFIB 7 OFB 9
130 Stefon Diggs Maryland SE 10 WR 20
131 Kurtis Drummond Michigan St FS 5 S 9
132 Lorenzo Doss Tulane NCB 6 CB 19
133 Ben Koyack Notre Dame TE 5 TE 5
134 Jamil Douglas Arizona St OG 9 OL 25
135 Christian Covington Rice 5T 5 IDL 14
136 Tyrus Thompson Oklahoma OT 13 OL 26
137 Josh Harper Fresno State SWR 3 WR 21
138 Jarvis Harrison Texas A&M OG 10 OL 27
139 Ibraheim Campbell Northwestern SS 5 S 10
140 Max Valles Virginia OFOB 3 OFB 10
141 Hayes Pullard USC OFIB 8 OFB 11
142 Ramik Wilson Georgia OFIB 9 OFB 12
143 Cameron Artis-Payne Auburn RBF 12 RB 13
144 Chris Hackett TCU FS 6 S 11
145 Nick O’Leary Florida St TE 6 TE 6
146 Antwan Goodley Baylor SWR 4 WR 22
147 Zach Hodges Harvard EDGE 18 EDGE 18
148 Kenny Bell Nebraska FL 9 WR 23
149 Durell Eskridge Syracuse FS 7 S 12
150 Rakeem Nunez-Roches Southern Miss NT 6 IDL 15
151 Shaquille Mason Georgia Tech OC 5 OL 28
152 B.J. Finney Kansas St OC 6 OL 29
153 Corey Robinson South Carolina OT 14 OL 30
154 Jesse James Penn State TE 7 TE 7
155 Adrian Amos Penn St FS 8 S 13
156 Vince Mayle Washington State FL 10 WR 24
157 Jake Ryan Michigan OFIB 10 OFB 13
158 Geneo Grissom Oklahoma EDGE 19 EDGE 19
159 Mike Hull Penn State OFIB 11 OFB 14
160 Davis Tull UT-Chattanooga EDGE 20 EDGE 20
161 Ellis McCarthy UCLA NT 7 IDL 16
162 Quandre Diggs Texas NCB 7 CB 20
163 Darius Philon Arkansas 3T 5 IDL 17
164 Dres Anderson Utah SE 11 WR 25
165 Martrell Spaight Arkansas OFOB 4 OFB 15
166 Tyeler Davison Fresno St NT 8 IDL 18
167 Dezmin Lewis Central Arkansas SE 12 WR 26
168 MyCole Pruitt Southern Illinois TE 8 TE 8
169 Sean Hickey Syracuse OT 15 OL 31
170 Cedric Reed Texas EDGE 21 EDGE 21
171 Mitch Morse Missouri OT 16 OL 32
172 Sean Mannion Oregon St QB 6 QB 6
173 Taiwan Jones Michigan State OFIB 12 OFB 16
174 Lynden Trail Norfolk St EDGE 22 EDGE 22
175 Blake Bell Oklahoma TE 9 TE 9
176 Josh Robinson Mississippi State RBF 13 RB 14
177 Craig Mager Texas State OCB 14 CB 21
178 Matt Jones Florida RBF 14 RB 15
179 Frank Clark Michigan EDGE 23 EDGE 23
180 Clayton Geathers Central Florida SS 6 S 14
181 Tony Washington Oregon OFOB 5 OFB 17
182 Malcolm Brown Texas RBF 15 RB 16
183 Karlos Williams Florida St RBF 16 RB 17
184 Joey Mbu Houston NT 9 IDL 19
185 Corey Crawford Clemson EDGE 24 EDGE 24
186 Ladarius Gunter Miami FL OCB 15 CB 22
187 JaCorey Shepherd Kansas OCB 16 CB 23
188 Jalston Fowler Alabama FB 1 RB 18
189 Titus Davis Central Michigan FL 11 WR 27
190 Robert Myers Tennessee State OG 11 OL 33
191 Derrick Lott Chattanooga 3T 6 IDL 20
192 Nick Marshall Auburn OCB 17 CB 24
193 Jean Sifrin Massachusetts TE 10 TE 10
194 Xavier Williams Northern Iowa NT 10 IDL 21
195 Bryce Hager Baylor OFOB 6 OFB 18
196 Louis Trinca-Pasat Iowa 3T 7 IDL 22
197 Max Garcia Florida OC 7 OL 34
198 Darren Waller Georgia Tech SE 13 WR 28
199 Justin Coleman Tennessee NCB 8 CB 25
200 Bobby McCain Memphis NCB 9 CB 26
201 Wes Saxton South Alabama TE 11 TE 11
202 Jordan Richards Stanford SS 7 S 15
203 Kyle Emanuel North Dakota St EDGE 25 EDGE 25
204 Austin Hill Arizona FL 12 WR 29
205 Bobby Richardson Indiana 3T 8 IDL 23
206 Nick Boyle Delaware TE 12 TE 12
207 Terrence Magee LSU RBF 17 RB 19
208 E.J. Bibbs Iowa St TE 13 TE 13
209 Austin Shepherd Alabama OG 12 OL 35
210 DeAndre Smelter Georgia Tech SE 14 WR 30
211 Adam Shead Oklahoma OG 13 OL 36
212 Leterrius Walton Central Michigan NT 11 IDL 24
213 Xzavier Dickson Alabama EDGE 26 EDGE 26
214 Mark Glowinski West Virginia OG 14 OL 37
215 Rannell Hall UCF SE 15 WR 31
216 Shaquille Riddick West Virginia EDGE 27 EDGE 27
217 Tyler Varga Yale FB 2 RB 20
218 Devante Davis UNLV SE 16 WR 32
219 Damian Swann Georgia OCB 18 CB 27
220 Jamon Brown Louisville OT 17 OL 38
221 Deontay Greenberry Houston SE 17 WR 33
222 Andrew Donnal Iowa OT 18 OL 39
223 Anthony Jefferson UCLA SS 8 S 16
224 Amarlo Herrera Georgia OFIB 13 OFB 19
225 Kaleb Eulls Mississippi St 3T 9 IDL 25
226 Terry Poole San Diego St OT 19 OL 40
227 Alani Fua BYU OFIB 14 OFB 20
228 Trey DePriest Alabama OFIB 15 OFB 21
229 Shane Carden East Carolina QB 7 QB 7
230 Deion Barnes Penn State EDGE 28 EDGE 28
231 Justin Cox Mississippi St FS 9 S 17
232 Ryan Russell Purdue EDGE 29 EDGE 29
233 Travis Raciti San Jose St 5T 6 IDL 26
234 Imoan Claiborne Northwestern St NCB 10 CB 28
235 Greg Mancz Toledo OC 8 OL 41
236 Kyshoen Jarrett Virginia Tech SS 9 S 18
237 John Crockett North Dakota St RBF 18 RB 21
238 Martin Ifedi Memphis EDGE 30 EDGE 30
239 Dominique Brown Louisville RBF 19 RB 22
240 Rory Anderson South Carolina TE 14 TE 14
241 Mario Alford West Virginia SWR 5 WR 34
242 Jeff Luc Cincinnati OFIB 16 OFB 22
243 Trey Williams Texas A&M RBC 2 RB 23
244 Miles Dieffenbach Penn St OG 15 OL 42
245 Tayo Fabuluje TCU OT 20 OL 43
246 Jon Feliciano Miami (FL) OG 16 OL 44
247 Davaris Daniels Notre Dame FL 13 WR 35
248 Quinton Spain West Virginia OG 17 OL 45
249 Chaz Green Florida OT 21 OL 46
250 Ray Drew Georgia 5T 7 IDL 27