:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/17032153/131661136.0.jpg)
Now it's time for me to share the results of applying the individual model I created last time to an out of sample population.
My out of sample QB seasons were taken from the 2007-09 seasons, the three before my sample. Ordinarily I like to take a sample and cut it unto to random split halves, one for modeling and the other for out of sample tests. That way effects like weather or a particularly bad flu season are controlled for. In this case I didn't because I didn't even want to have the 2007-09 data on my hard drive. I knew I'd run into sample issues and I didn't want to tempt myself to work with my other half. Even just the tip, just to see how it felt.
The idea of modeling injuries like this is suspect even for a stat crazy fan like me and for that reason I wanted to be as rigorous as possible with the parts that could help refute the model. I want it to work - because of the time I've put into it and general nerdiness - and I really didn't want that to become a self fulfilling desire.
So I ran the Kolmogorov-Smirnov test against the new data, using the population average model for both the individualized models and the population average model. Again I used the null hypothesis that the actual injuries and simulated injuries were from the same population. Higher p values mean greater odds that the null hypothesis is true.
The population average model had a p value of .83 - pretty good and better than the 20010-12 seasons.
The individualized model had a p value of, I shit you not, .999 - that's higher than the p value for the sample I prepared the model with.
When I got that result I opened a celebratory beer and forgot to drink it as I checked my code.
What does this mean? Well let's not get too excited. The individualized model I created describes actual injury report listings better than assuming every QB is league average. Simulated model results are more or less indistinguishable from the actual season outcomes. Because injury report listings predictably describe actual injuries the model can be extrapolated as representative of actual injuries.
My default position is to assume all players have league average injury risk absent compelling arguments to assume otherwise. My position hasn't changed, I've just added the caveat, that QB injury risk should be individualized according to the my risk model unless compelling information suggests it is inaccurate in a specific case.
In the days and weeks to come I will be running further tests to find the limits, holes, and possible refinements of this model. They certainly exist
In the mean time, can the model do what we all want it to and predict future injury risk?
The answer to that question comes down to whether we can predict sacks per game and rush yards per game. We know we can, so the answer is yes. The interesting question is how long does a QB need to be in the league before those two values are useful? The answer is about a season for both - shorter for rush yards per game, longer for sacks. Though, to be honest with you, high rush QBs probably take longer.
So let's look at my check list from the last article:
- Does the describe out of sample data well? Yes.
- Can the model be used to predict ? Yes.
- What should it be used for? . . .
I have two immediate thoughts. First, there is some anecdotal evidence that QBs make a tradeoff decision between sacks and interceptions. With a model that predicts injuries based, in part, on sack totals it is possible to make a prediction of the optimal solution to that trade off. Some people say the 49ers coaching staff told Alex Smith to take more sacks to limit interceptions. If they did then we can see to what extent it was a good idea and to what extent that decision may have contributed to his injuries.
But I'll let the sack/interception investigation sit for now so that my veiled and unsupported accusation against the 49ers FO can work its evil purpose.
My second thought is let's take a look at whether rushing QBs make up for their higher injury rates in added value.
Okay, that wasn't my second thought. It was my first. It's transparently why I did this whole thing.
Before we have this conversation I want to be perfectly clear that what follows isn't particularly rigorous. I'm going to be doing a fair amount of hypothetical play mix hanging and I'm not going to change the play efficiencies when I do. That's wrong. If Wilson threw more it would likely be for less yards per attempt. At times it may take some mental gymnastics to intuit the impact of that effect but it is important to keep in mind.
First let's imagine a world were Wilson always gets 30 attempts per game. They can be divided up between runs and passes any way we chose. I know that's not how it works but bear with me. Here's a chart of injury risk possibilities between 0 runs and 10 runs per game:
The green line at 6 is last season. Note that the highest value, 10 has a lower mean games missed - that is the model breaking - not an actual effect.
There are a few reasons this chart is silly. First, his YPC wouldn't stay the same, second some of the new runs would be sacks might lead to a different sack rate. Finally this chart replaced carries with pass attempts - it might be more realistic to replace QB carries with RB carries. What it demonstrates though is that more yards per game tracks to more injury risk.
In order to decide if the runs are worth it we have to do a cost-benefit analysis. That's very tricky with football plays. The reality is we want Wilson to throw more than he does because it's more productive yard wise than running - even considering the increased risk. He's a good throwing QB. Generally the plays where Wilson runs it's either a designed run with a real coaching purpose or he's doing it because the throw is too risk or impossible on that particular play.
Unfortunately I have no way to tell the difference. That means that I can't tell if the choice was Wilson run v. Wilson sack/throw-away or Wilson run v. Lynch run except by referencing the stats game tape nerds keep.
They tell me Wilson is 39% designed rush and 61% scramble.
Given that, I'm going to say that 39% of rushes taken away from Wilson will be given to Lynch and 61% will become pass attempts for 0 yards with average interception risk. Following is a chart showing the expected production of Wilson's 30 plays per game given different run totals and no injuries. You'll also notice a few curved lines - that is the yards expected per game from the QB position. Basically I do all the Wilson math, figure how many games (fractions of games) we expect him to miss for each run total and then replace his values with Tarvaris Jackson's for those. The point where the value starts to come down is where the value of the QB runs is outweighed by the mean injury risk.
Other than assuming that all the plays have the same efficiency regardless of play call mix my least favorite assumption made in that graph is that the throws made in the place of scrambles would result in 0 yards. A lot of them would as throw aways, but guys, Wilson is pretty damned good at getting yards with his arm.
And it is an important assumption. Even small yardages from throwing instead of running result in a dramatic decrease in the relative usefulness of QB carries. So in the interest of full disclosure here are plots for 0 (same as above), 1, 2, 3, and 4 yards gained on those plays. The numbers in the legend are the yards per scrambled throw and then the yardage maximizing number of runs. Note 0 yards was the initial assumption and is the same curve as in the plot above.
Basically, according to this model, Wilson is sitting pretty in an area that maximizes the value of his runs balanced against injury risk. On the field he doesn't have a calculator. The stuff I get a computer to do he has to use muscle memory, eyes, intelligence, and trained intuition. And the evidence is tacking up that he is very good at putting those things to use.
Like I said earlier I doubt I'm done with this model. Expect this trilogy to become increasingly misnamed.