:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/35452050/463882399.0.jpg)
In all the recent discussion of QB rankings, I have seen several conversations pop up about total production, utilization, wins, QBR, ANY/A, DVOA, EPA, WPA, and the old NFL passer efficiency rating. The first flicker of this fanpost popped up in some friendly banter with the amicable denizens of NinersNation.com, and then the light bulb really went on responding to criticism of a poor Aaron Rodgers game where he had a QBR of 33.3, which is bad (QBR was right in this case). Anyway, I used the NFCCG as an example of QBR gone wrong. Read on to get your July football fix...
----------
When I watched the NFCCG, my immediate impression was that Wilson played with Typical Wilson efficiency, and although Colin Kaepernick almost won the game with his legs, he lost the game with his arm. Here are their stat lines from the NFCCG and a comparison of their QBR and ANY/A:
Kaepernick: 14/24, 153 yards passing, 1 TD, 2 INT, 2 sacks/6 yards, 11 rushes, 130 yards rushing, 1 lost fumble
Wilson: 15/25, 215 yards passing, 1 TD, 0 INT, 4 sacks/22 yards, 5 rushes, 0 yards rushing, 1 lost fumble
QBR: Kaepernick 65.1 (I hate that guy) vs. Wilson 38.9 (WT?$*%!#?)
ANY/A: Kaepernick 2.96 (cringeworthy) vs. Wilson 7.34 (elite)
Wins: Kaepernick 0 ('nuff said) vs. Wilson 1 (doomo arigatoo Mr. Roboto!)
Obviously, some lunatics will point to QBR as proof that Kaepernick played better, while those legally of sound mind will point to ANY/A as all the proof necessary to predict Wilson leading his people to the promised land a short two weeks later. Jokes aside, Wilson had a better game as a passer and Kaepernick had a better game as a runner, but the summation of their running and passing seemed a lot closer than either QBR or ANY/A would suggest. Let's look at the discrepancy between QBR and ANY/A a little more thoroughly.
I'll start with ANY/A because it's very straight forward. ANY/A stands for Adjusted Net Yards per Attempt, and it is easily calculated as:
ANY/A = [(passing yards - sack yards + 20 * TD passes - 45 * INT) / (passing attempts + sacks)].
In terms of evaluating passer efficiency, ANY/A is considered by most to be superior to the old passer efficiency rating, and the value is quite intuitive - average yards per pass play (adjusted for sacks, TDs, and INTs). ANY/A doesn't incorporate rushing yardage or fumbles.
QBR is completely proprietary (although some details are released), and has very high correlations with EPA, WPA, future QBR, and future wins. There have been some issues with acceptance of QBR, and the example above in the NFCCG is not the only instance of discrepancy between what we know is a good QB performance and what QBR tells us is. From the wikipedia page on QBR:
Total QBR system gave the Denver Broncos' Tim Tebow a higher rating than the Green Bay Packers' Aaron Rodgers in their respective Week 5 contests in 2011. Noting that Rodgers completed 26 of 39 passes for 396 yards and two touchdowns in a win over the Atlanta Falcons, while Tebow [pqlqi note: in one half of football] completed four of 10 passes for 79 yards and a touchdown, and six rushes for 38 yards and a touchdown, in a loss to the San Diego Chargers
Paraphrasing from ESPN.com's page on explaining QBR, QBR is a model designed to evaluate the contribution of a QB to winning, and is built on the prior 10 years of NFL data. It divides credit on pass plays between pass protection, the throw (air yards), the catch, and run after catch. Subjectivity is introduced regarding pass protection, accuracy, and definition of a drop by the WR. Air yards and YAR (yards after reception) are completely objective.
QBR also initially incorporated a fairly heavy clutch factor, but it has been partially nullified as it was unacceptably skewing results. Normal plays have a clutch factor of 1.0, plays in high leverage situations at the end of the game have a clutch factor of 1.0 (previously as high as 3.0), and plays at the end of a blowout game are given a clutch factor as low as about 0.3.
The clutch factor is used to weight the play in the context of the game, so an INT thrown early in the game will have a weighting of 1.0, whereas an INT in the last ten minutes of a blowout will have a much lower weighted value near 0.3. The clutch factor unfairly penalizes QBs (particularly those on bad teams) if their team falls behind big early in the game, as their actual success can be reduced by as much as 70% in the QBR formula.
Regarding other components of QBR, this is an excerpt taken directly from the ESPN page:
On most other plays, quarterbacks receive some portion of credit for the result of the play, including defensive pass interference, intentional grounding, scrambles, sacks, fumbles, fumble recoveries (Carson Palmer once recovered a teammate's fumble that saved the game for the Bengals) and throwaways. On plays when the QB just hands off to a running back, we didn't assign any credit to the QB.
While QBR is a black box, we know it separates air yards from YAR (good), it tries to separate incompletions into drops vs inaccurate throws but is limited by observer interpretation, and awards value/penalty for scrambling, drawing DPI, taking sacks, fumbling, recovering fumbles, and committing penalties. And it penalizes QBs on teams that fall behind by a large margin.
The major failing of ANY/A is that it doesn't incorporate rushing ability into quarterback value, thus neglecting significant contributions of the new QBs-who-happens-to-be-an-athlete. The major failing of QBR is that it's weighting of running can make a bad passing performance look good, or even great, to the casual observer who is ignorant of these subtleties (which aren't so subtle) - see the NFCCG and the Rodgers/Tebow examples above.
So what if we added rushing stats to the ANY/A equation - it'd have the main benefits of QBR without the wacky weighting, subjective "stats", or the proprietary measures. It would give credit where credit is due to game changing scrambling performances but hopefully not exaggerate the value of good rushing relative to good passing. Let's call it TANY/A, Total (rushing and passing) Adjusted Net Yards per Attempt. We'll add rushing and passing yards and subtract sack yards, adjust the totals for TDs and turnovers, and divide the batch by the QBs rushing and passing plays.
For the purposes of this first model, I will treat fumbles as half of an interception (According to Football Outsiders, fumbles tend to be less costly than interceptions, but fumbles by QBs are largely behind the LOS, making QB fumbles more costly than average fumbles; for simplicity, I am using the 2 fumbles = 1 INT conversion).
TANY/A = [ {passing yards + rushing yards - sack yards + 20 * (TD passes + TD rushes) - 45 * (INT + 0.5 * Fumbles) } / (passing attempts + sacks + rush attempts) ].
Applying this formula to the NFCCG, we get:
TANY/A: Kaepernick = 4.37 vs Wilson = 5.24, which actually seems about right.
If we apply this to the 2013 league year, for QBs who started 8 games or more, we get the following results:
| ANY/A | TANY/A | QBR | |
|---|---|---|---|
| Nick Foles | 9.2 | 8.4 | 69.0 |
| Peyton Manning | 8.9 | 8.1 | 82.9 |
| Philip Rivers | 7.8 | 7.4 | 71.7 |
| Aaron Rodgers | 8.0 | 7.3 | 68.7 |
| Drew Brees | 7.5 | 7.1 | 70.5 |
| Russell Wilson | 7.1 | 6.4 | 58.9 |
| Colin Kaepernick | 5.0 | 6.3 | 68.6 |
| Tony Romo | 6.5 | 6.2 | 59.5 |
| Andrew Luck | 6.1 | 6.0 | 62.0 |
| Jay Cutler | 6.2 | 5.9 | 66.4 |
| Ben Roethlisberger | 6.2 | 5.9 | 54.3 |
| Andy Dalton | 6.3 | 5.9 | 55.8 |
| Matthew Stafford | 6.4 | 5.8 | 52.5 |
| Cam Newton | 5.7 | 5.7 | 56.2 |
| Alex Smith | 5.9 | 5.6 | 49.4 |
| Tom Brady | 6.1 | 5.6 | 61.1 |
| Matt Ryan | 5.7 | 5.5 | 61.1 |
| Ryan Fitzpatrick | 5.6 | 5.2 | 55.4 |
| Carson Palmer | 5.7 | 5.2 | 51.9 |
| Jason Campbell | 5.3 | 5.2 | 38.6 |
| Robert Griffin | 5.5 | 5.1 | 40.1 |
| Case Keenum | 5.4 | 5.0 | 34.5 |
| Ryan Tannehill | 5.0 | 4.8 | 45.8 |
| Chad Henne | 4.9 | 4.7 | 31.9 |
| Kellen Clemens | 5.3 | 4.5 | 38.2 |
| Christian Ponder | 4.8 | 4.5 | 51.2 |
| Mike Glennon | 6.7 | 4.4 | 45.6 |
| EJ Manuel | 4.9 | 4.4 | 42.3 |
| Matt Schaub | 4.5 | 4.4 | 37.3 |
| Terrelle Pryor | 4.1 | 4.4 | 30.5 |
| Joe Flacco | 4.5 | 4.3 | 46.7 |
| Eli Manning | 4.6 | 4.2 | 36.5 |
| Geno Smith | 4.2 | 4.2 | 35.9 |
Peyton Manning and Nick Foles stand out as the best performers in the league in 2013 (the fact is, Foles had a phenomenal season and we all know about the caveats, let's not waste time on that specific discussion). There is a clear second tier of Drew Brees, Aarron Rodgers, and Phillip Rivers. Then we fall into a pretty linear distribution of the remainder of the league where there are no readily identifiable tiers. At the top of the "rest of the league" group, Kaepernick and Wilson provide another interesting insight.
Russell is a fair bit ahead in ANY/A, 7.1 vs 5.0, but given small edges to Kaepernick in ball security, less sack yards, and more rushing TDs, they end up neck and neck on the season in TANY/A, with Wilson and Kaepernick at 6.4 and 6.3, respectively (if turnovers were weighted a fraction heavier, Kaepernick would get the edge).
I feel like Matt Ryan and Tom Brady are lower on the list than their outright abilities and their history in the league would suggest, and it's almost certainly due to the lack of decent targets on both teams (sorry Roddy, age and injury have taken their toll) and a lack of a decent rushing attack in Atlanta. The rest of the list seems to fall nicely in line.
I know this is untested for win probability purposes. I know QBR is one of the best stats for predicting future success, and QBR differential is a fantastic predictor of wins. I'll leave it to the statistical phenoms around here to sort it out.