Editors note: As a byproduct of dealing with some really useless SBN publishing issues this morning, I managed to lose the original piece. This is just a re-posting with some minor changes as some of the content was lost.
On Friday, we went through a look at 2008 NFL team efficiency with respect to cap management. Football Outsiders devised a quick metric to evaluate teams on their ability to win with a lower payroll as well as their ability to roll over cap room for next year. That lead to a brief discussion in the comments as to whether or not salaries mean very much with regards to success/talent. Because there is a salary cap, major spending discrepancies are, in theory, wiped out.
For salary data, the handy USA Today NFL Salaries database.
For standings and point differential data, Pro Football Reference.
For Team Efficiency DVOA data, Football Outsiders.
Win % and Point Differential are both flawed in the effort to determine true team talent. DVOA is about the best metric available and should be the more reasonable evaluation of performance.
Because I wanted to compare data from 2002-2008 (the Texans came in for the 2002 season), a straight comparison of winning percentages, point differential, or DVOA to salary would be skewed by upward trending average payroll. As such, I normalized season salaries with something akin to Salary+ (if you are familiar with baseball statistics): Adjusted Salary is then each individual team salary in any given year divided by the average salary for that respective year. Simple enough. I then dropped all the data into Minitab and ran three separate regressions.
| Adjusted Salary Regression Results | |||||||
|
|
|
|
|
||||
| Win Percentage to Adjusted Salary |
|
|
|
||||
|
|
|
|
||||
|
|
|
|
Both Win % and Point Differential are statistically significant with respect to Adjusted Salary. While they are significant, the correlation is dreadful: check the scatterplots below. For the two flawed statistics (Win% and Point Diff), there is a minor amount of correlation to go along with the significant p-value and minuscule adjusted r-squared. DVOA is just not affected by Adjusted Salary.
How can we justify the fact that the outcomes measures show significance while the more accurate team talent measure has none? We have a fairly large sample size, with n=224. Even with low correlation, large enough data pools can prove statistical significance in weak relationships. The rest is just noise. In this case, it's safer to look at the adjusted r-squared values to know that only a fraction of our variation is explained by the model. This is a case of statistical significance without practical significance.
There may be a relationship between success and payroll, but that relationship is hazy at best. As long as a GM is spending money at a level near the salary cap, the rest has to do with how that money has been spent (surprise!). Does the result make logical sense? Yup. There are enough bad contracts, injuries, and inefficiently run teams to negate the effects of slightly differing payrolls. I'm sure you can think of enough individual data points (2008 Seahawks, the most recent Redskins teams, Oakland, etc.) to understand the concept. The data backs it up.
Scatterplots of the data (you can click on any of the scatterplots for the full image):
A few other notes on salary differential that I came upon:
- The largest salary disparity in any single season came in 2002, where the Atlanta Falcons had 196% the payroll of the Vikings ( $85.1 million to $43.5 million)
- The largest single Adjusted Salary? The 2003 Redskins, at 140.1
- The 2002 Vikings had the lowest Adjusted Salary at 66.0
- For recent reference, the Raiders had the highest Adjusted Salary at 134.5 last year while the Chiefs brought up the bottom at 73.8