While everyone is busy breaking down the scrimmage film with a Jim Garrison-like passion, I thought I would sneak a little preseason preview of some concepts I have been thinking about for how to measure success on a down by down basis. If you want to avoid the nerdy details, skip down for some pretty charts.
Looking at down by down success is a tricky thing and right now there are only limited tools for how to evaluate how an offense is utilizing its most precious resource. The only mainstream tool is third down conversion percentage. This tool’s simplicity is both its weakness and a hidden strength.
Third down conversion rate does not take into consideration how hard your third downs are to convert. Two teams could have identical conversion percentages but if one team has a lot of third and shorts and the other doesn’t the team that doesn’t is accomplishing a much tougher job than the first team. That absence of context is also the hidden strength. Third down percentage isn’t a great predictor of how good your team performs on third down as much as it is an all-encompassing look at how good your team is at getting to manageable third downs and then converting them.
The newer stat that looks at all downs is the Success Rate metric, one I have been on record as not being a huge fan of. Success Rate is a more nuanced look at each down and assigns them a binary pass fail grade depending on whether they meet certain threshold criteria. A binary makes some sense on third down and more sense over the collection of downs, but there is too much opportunity for other value to come and go for the binary to be of major use.
A third way is an expected value (EV). How much value is each team adding or subtracting on given downs. This is a literal value look at ranking teams by what they are accomplishing on a given downs. I have traditionally used this metric but again, it lacks the detail of what is really going on behind the numbers. An EV look tends to lend a lot of value to big play teams and punish consistent gainers. There is evidence to support the rankings coming out that way, but again, I don’t think the numbers tell a good football story in one dimension.
The Early Downs Breakthrough
As I began digging into this I pulled all kinds of numbers looking at each of the three downs separately before it dawned on me, first and second down are really a package deal. They are the offense’s opportunity to either do something big or maximize their chances of a third down conversion, first and second downs and typically on the offense’s terms. You can only create big plays so often and even being good at getting in great third downs all the time still means you are having a lot of plays with a chance for the defense to get off of the field. 3rd and 1’s are converted 72% of the time by the offense, so if you get in three of those situations the odds are nearly two to one that you get stuffed on one of them. Being good at avoiding third downs is a better skill for an offense than getting in manageable ones (although both are obviously preferred).
So to that end, I put together three key metrics for an offense for 1st and 2nd downs:
Early Conversion %: Percent of first downs that are created prior to third down. An average team will convert at about 50% with the best offenses closing in on 60%, like the 2011 Oregon offense.
Bonus Yards: This is a big play metric. For the plays that create a conversion, how many yards beyond the sticks does the average play go. Average teams are around 6.5. Mike Leach’s 2005 Texas Tech team was one of the best ever at 9 yards beyond the stick.
Average 3rd Down Distance: The first two metrics are about the successes, historically, most football coaches are more about minimizing the negative. This metric is for them. For the 50% of the time that the average team faces a third down, how many yards are they typically facing. The average team still has 6.5 yards to go on an average third down. Last year’s Air Force team that Michigan faced was the best of the last 10 years with an average distance faced of 4.0 yards for the season.
Third Down
Now that early downs have hopefully been understood a little better, it’s time to look at third down and focus on a true measure of the down itself. One option that’s sometimes used is to break down the conversion rates into yardage buckets representing short yardage, medium, etc. This isn’t the worst way to go about it, but still isn’t great. Unless its over a large portion of time, sample size problems are likely and you still potentially have problems, although much smaller now, of where do the actuals trials fall into the buckets. Too many buckets and the splits become hard differentiate, too few and there is little continuity to what you are measuring.
To try and solve these issues, here is my suggested stat:
Adjusted 3rd Down Conversion Percentage: Each third down distance has an average conversion rate that looks like this:
1 yard to go converts at 72%, 10 yards to go at 28%. If an offense converts a third and 1, they get +28% for that play. Fail and it’s –72%. Average up all the third downs for a period and you are left with a single number to reflect how a team has done on third downs, that isn’t weighted by being better at first and second down. The other nice thing is that it is naturally anchored to zero. An average team is at +0%. 2011 Wisconsin with Russell Wilson and Montee Ball was the best Big Ten third down team at +16%. 2011 Alabama was the best third down defense at –15%.
2012 Results
Taking all the above analysis, I pulled the results for last season and put them together in a fancy new Tableau table (click to control the view [ed-S: we know; we're working on the links]).
2012 Offense
Circle size represents average third down distance
So, Michigan was pretty good on a down by down basis, last year. Only Clemson and A&M where better at third downs when accounting for yards to go. Michigan was also one of the best teams at avoiding third downs altogether, converting on first or second down about 54% of the time.
The other big take away from this is that there are a lot of Big Ten teams at the left hand side of the chart. It’s a bit hard to tell from this view, but Big Ten teams are some of the best at managing third down distance but some of the worst at everything else. Fully half of the teams in the conference are in the lower left quadrant of teams that are bad at both. An offense whose goal is to get into manageable 3rd downs is an offense that is set up to fail.
2012 Defense
Michigan lands pretty average across the Big Five conference landscape in both early downs and third downs on the defensive side. The strength of Michigan State’s defense really shows up here, as they only allowed teams to convert before 3rd down about 2 out of 5 times.
I am trying to put together a package of weekly reports and rankings that I can publish online. If anyone has any thoughts as to what you want to see that aren’t otherwise available, I am open for suggestions.
I think these charts do a good job of reflecting what’s happening on a down by down basis. What they don’t show are the impact of big plays and high leverage plays like turnovers and red zone plays.