When Ace was scouting Kai-Leon Herbert, the offensive tackle prospect who announced his commitment to Michigan this week, we took note of the big disagreement in his rankings:
| Scout | Rivals | ESPN | 247 | 247 Comp |
4*, #22 OT, | 4*, #10 OT, | 4*, 83, #15 OT, | 3*, 86, #72 OT, | 4*, #22 OT, |
Some variation for a project recruit with big upside isn't that weird, but one site having him threatening the top 50 while another has him barely among the top 700 is some serious disparity. BiSB even wondered aloud if big disagreements like that portend anything for a guy. And well, I have a database. Let's see.
Methodology
I'm going back to my STARs ratings, which are a composite of the four main recruiting sites' scoring systems/stars/rankings normalized to a sliding scale of five stars. I cut out specialists, then used only players for whom we have at least three rankings to go from, and ran a standard deviation.
So What Happens When They Agree?
This wasn't very useful because most of the guys with high agreement were very well scouted (duh) and a few were like the toppomost of the tippytop. There were 12 guys who sparked almost total agreement (ordered by rating):
| Player | Class | Rivals | Scout | ESPN | 247 | STDev |
|---|---|---|---|---|---|---|
| Rashan Gary | 2016 | 5.0 | 5.0 | 5.0 | 5.0 | 0.00 |
| Jabrill Peppers | 2014 | 5.0 | 5.0 | 5.0 | 5.0 | 0.00 |
| Ben Bredeson | 2016 | 4.6 | 4.6 | 4.6 | 4.6 | 0.00 |
| Justin Boren | 2006 | 4.6 | 4.6 | 4.6 | 0.00 | |
| Chris Fox | 2013 | 4.2 | 4.2 | 4.2 | 4.2 | 0.00 |
| Josh Ross | 2017 | 4.0 | 4.0 | 4.0 | 4.0 | 0.00 |
| M. Witherspoon | 2008 | 4.0 | 4.0 | 4.0 | 0.00 | |
| Ben Braden | 2012 | 3.6 | 3.6 | 3.6 | 3.6 | 0.00 |
| Christian Pace | 2010 | 3.6 | 3.6 | 3.6 | 0.00 | |
| Teric Jones | 2009 | 3.6 | 3.6 | 3.6 | 0.00 | |
| Frank Clark | 2011 | 3.4 | 3.4 | 3.4 | 3.4 | 0.00 |
| J.T. Floyd | 2008 | 3.2 | 3.2 | 3.2 | 0.00 |
Teric Jones, Christian Pace, and Chris Fox lost their careers to injury, but after that only Marcus Witherspoon (for off-field reasons) didn't end up a regular starter, pending the careers of Bredeson (who may be the first tackle in this year) and Rashan Gary (who may be the greatest tackle in years).
So What Happens If They Disagree?
Here's the 15 biggest disparities. I've highlighted the biggest outliers.
| # | Player | Class | Rivals | Scout | ESPN | 247 | STDev |
|---|---|---|---|---|---|---|---|
| 1 | John Ferrara | 2006 | 3.4 | 3.2 | 1.8 | 0.90 | |
| 2 | Jason Kates | 2006 | 4.0 | 2.3 | 3.2 | 0.88 | |
| 3 | Austin White | 2010 | 3.6 | 4.0 | 3.4 | 2.0 | 0.87 |
| 4 | Kevin Koger | 2008 | 4.2 | 4.2 | 3.0 | 0.69 | |
| 5 | Conelius Jones | 2010 | 3.4 | 2.8 | 3.4 | 2.0 | 0.67 |
| 6 | William Campbell | 2009 | 4.8 | 4.6 | 3.6 | 0.64 | |
| 7 | Junior Hemingway | 2007 | 3.6 | 3.8 | 4.8 | 0.64 | |
| 8 | Brandon Smith | 2008 | 4.2 | 4.4 | 3.2 | 0.64 | |
| 9 | Chase Lasater | 2017 | 3.4 | 2.8 | 3.2 | 2.0* | 0.62 |
| 10 | Brandon Moore | 2008 | 4.0 | 3.2 | 4.4 | 0.61 | |
| 11 | Isaiah Bell | 2009 | 3.6 | 3.2 | 4.4 | 0.61 | |
| 12 | Mark Huyge | 2007 | 3.4 | 2.5 | 2.3 | 0.60 | |
| 13 | Rocko Khoury | 2008 | 3.6 | 3.4 | 2.5 | 0.59 | |
| 14 | Boubacar Cissoko | 2008 | 4.6 | 4.6 | 3.6 | 0.58 | |
| 15 | D.J. Williamson | 2010 | 3.2 | 2.5 | 3.6 | 0.56 |
*247 hasn't ranked Lasater yet
Big winners on there are Koger, Hemingway, and Huyge, though BWC turned out okay once Hoke got his hands on him. FWIW the one guy Scout and Rivals really disagreed on before ESPN entered the ring was Alex Mitchell. Meanwhile I had to go back to the blogspot page to find Brian's take on the huge disparity over Junior Hemingway:
So, yeah... those numbers above disagree fiercely. Hemingway is either in the top 10, 20, 30, or 40 receivers in the country, depending on who you listen to. Rivals went so far as to downgrade him to a three-star after season's end for reasons unknown (read: plain old provincialism on the part of that particular region's rankings guru). Meanwhile, ESPN is freakin' out over here. Scout and Creepy Tom Lemming split the difference.
Ironically he turned out exactly as advertised:
Leaving aside his exact proportions of shirtlessness for the moment, Hemingway is a leaper capable of ridiculous grabs. His overall athleticism has been questioned by those skeptical of his talent, but no one debates his body control, leaping ability, and hands.
The most noticeable thing other than how many of those guys didn't pan out was that ESPN was usually the oddball.
How Do the Sites Compare?
Was ESPN always so odd? They ranked Koger as a DE (Brian did posit at the time that a move to DE was likely, since Michigan had few), while Rivals and Scout had him the #4 or #6 tight end. But it came up enough I had to look at them versus the average to see if that was normal:
(click makes big)
Mathematically (by deviation of squares) they were by far the most likely to disagree with their peers:
ESPN: 39.73
Scout: 31.74
Rivals: 24.09
247: 13.80
If they were highly accurate that would be interesting, but as you see by the outliers, only one of the dudes they seemed super-way-excited about even started (though Metellus has time).
It was also interesting to see which players each site was most panting/skeptical about. I'll highlight if they got it right:
WHEN THE SITES ARE BEARISH:
| Rivals | Scout | ESPN | 247Sports |
|---|---|---|---|
| Nolan Ulizio (-0.5) | Jason Kates (-0.9) | John Ferrara (-1) | Austin White (-1.3) |
| Bryan Mone (-0.5) | Brandon Moore (-0.7) | Kevin Koger (-0.8) | Conelius Jones (-0.9) |
| Jr Hemingway (-0.5) | D.J. Williamson (-0.6) | Brandon Smith (-0.7) | Jake Butt (-0.5) |
| Chris Wormley (-0.5) | Davion Rogers (-0.6) | Will Campbell (-0.7) | Jourdan Lewis (-0.4) |
| Patrick Omameh (-0.4) | Reuben Jones (-0.5) | Rocko Khoury (-0.7) | Devin Asiasi (-0.4) |
Some of those guys it's too early to tell. But I might be a bit more leery of Rivals skepticism and hoping Reuben Jones proves Scout can be wrong.
Meanwhile in high expectations, here are the guys certain sites thought would outperform the consensus of their peers:
WHEN THE SITES ARE BULLISH:
| Rivals | Scout | ESPN | 247 |
|---|---|---|---|
| Jason Kates (+0.9) | Austin White (+0.9) | Jr Hemingway (+0.7) | Chris Wormley (+0.6) |
| Mark Huyge (+0.7) | P.Omameh (+0.6) | Isaiah Bell (+0.7) | Nate Johnson (+0.5) |
| John Ferrara (+0.6) | Josh Furman (+0.5) | Brandon Moore (+0.5) | Dennis Norfleet (+0.4) |
| Greg Mathews (+0.5) | Sam McGuffie (+0.5) | Q.Washington (+0.5) | Erik Magnuson (+0.4) |
| Conelius Jones (+0.5) | (tie* +0.5) | Conelius Jones (+0.5) | Mason Cole (+0.4) |
* Marrell Evans, Brandon Smith, Tom Strobel, De'Veon Smith, and John Ferrara.
Some of the guys I didn't highlight were fine but only insomuch as they met their recruiting expectations. At least Rivals knew before everyone else that Huyge was unkillable but otherwise woooooof. Meanwhile Scout got burned by some major athletes (Furman and McGuffie at least wound up starting elsewhere), but the only real diamond they pointed out was Omameh; the five-way tie varied from slightly too positive (D.Smith, Ferrara) to vastly overrating (Evans, Strobel, B.Smith).
Of course these are just small sample sizes—useful for gauging extreme outliers but little else. So I used scatter charts to see if there was a major difference in the aggregate, tracking all their recruiting ratings by deviation from the mean and their starts/eligible seasons. The best scouting site would have the most bubbles very high and to the right, and fewest bubbles high and to the left (guys they were skeptical about who got a lot of starts).
A few major outliers got cut out but a picture has emerged. When Scout says a guy is good you should probably pay attention. Rivals has a low batting average but will connect as often as they whiff. ESPN appears to lose track of guys who aren't ranked at the very top, so their outliers may be more cautionary than anything. 247 plays it mostly safe but once in awhile takes a calculated risk that usually pays off.
Or that they're huge Norfleet fans. One understands.
What does that mean for Herbert and the OL this year?
We haven't seen this kind of distribution before, honestly. These rankings could change so much before February however that I wouldn't put much stock in them anyway. The Herbert disagreement doesn't look so bad in the STARs. With nothing else to go on, I'd say keep an eye on 247's rating to see if that jumps after the Opening, and otherwise trust that Scout has him pegged.