Taking a Schott at play calling
There is probably no subject of a discussion in Seattle sports more divisive currently than that of Brian Schottenheimer. The brand new to us Offensive Coordinator is loved, hated, blamed, credited, and discussed at length among the Seahawks reporting world. As the off-season slogs on, we hear a lot of criticism with regard to play calling just like his predecessor. But are the complaints against the newest pariah justified?
A brief look at the past
It's no secret that previous to his departure Darrell Bevell had a few bad seasons in a row. What once was a high powered offense leading the league in DVOA in 2015 had, though injury (2016), and a complete erosion of run efficiency (2017) fallen to nearly league average. Part of which is a credit to the toughness of Russell Wilson who, despite a grade 2 MCL sprain, never missed a contest under his former coordinator. While Seattle's overall offensive DVOA dipped sharply starting in 2016, in 2017 passing DVOA returned to a respectable level at 21.1%. Meaning Wilson was about 20% better than the average QB given game state and opponent adjustments on the year. Certainly down from the greater than 40% of 2015 but significantly better than his injured play of the 2016 season.
While we don't hear a lot about it now, a frequent complaint about Bevell was his lack of adaptation. Specifically his tendency to call "run-run-pass." As outlined in detail here, a lot of these complaints didn't actually hold much weight. But, perhaps we could be missing something. Accounting for game state can be a tricky thing and not every play call sequence is the same. One could argue that we should account for game state rather than looking at sequences in a vacuum.
Our neighbors save the day
One of the difficulties in judging play calling at the NFL level is defining "good." What, exactly, is a good play call in a given situation? That is largely influenced by scheme and personnel, so immediately we should expect some variance team to team. But nevertheless we should be able to come up with some guide for situational play calling using the other teams in the league.This is exactly what I propose for this little excursion.
K nearest neighbors is an algorithm of a few different flavors that tries to do exactly what we're talking about. In general we're going to take things like which half a team is playing in, down, distance to go, field position, and score differential of a play. Then we can see how often other teams in the K most similar positions acted. This should give us an idea of what the typical NFL team would do in a given situation in a given season.
Can you Bevellieve it?
Thanks to the nflscrapR folks we can take a look at the 2017 season above and look at what teams deviated most from their nearest neighbors. One way to look at this graph is the farther to the left a team is, the less they are deviating from how the league calls plays according to the 20 nearest plays given game state. That is, the more to the left the team is the more they follow conventional wisdom.
One of the teams that immediately jumped out at me was New England, and in fact you'll see them jump out again in 2018. But back to Bevell. Seattle was, in 2017, the least likely to buck the trends of their contemporaries given game state in general. But, the way this model is trained, being away from the other teams could mean a few things. A team could pass or run at an exceptionally high rate, or they could be unpredictable in general. So, for 2017 what was it?
Controlling for game state, Seattle was arguably the most "pass happy" offense in 2017 second only to New England. It would seem, given these results that much of the "run-run-pass" criticism of Bevell in 2017 was unfounded. But, perhaps there is something to be said about his predictability.
Using the same data set, I tested to see if you could predict 2017 play calling based on Bevell's previous years. Now, this is a pretty quick hack at it and maybe there are other significant factors, but using only seconds in half remaining, down, yards to go, and score differential a simple logistic regression predicts 2017 play calls 70% of the time.
As another approach I used all of the league's play calls from 2009-2016 to train the exact same model to predict the Seattle Bevell Sonics, and you basically get the same out of sample 2017 predictability. So, if you on your couch lamented that you could predict his play calling? Well, there might be enough here for me to believe you, with at least some level of confidence. Of course there is more to play calling than run versus pass, but barring the league charting data appearing in my inbox unsolicited, that's what we have for now.
Shoot your Schott already
I've spend way, way too much time talking about 2017 already. And you're probably as sick of it as I am. Look, I can't control where my mind wanders in the Rstudio console anymore than you can. We're all along for the ride.
A fair place to start is building the same kind of model for Schottenheimer, so I did. First, I looked at if we could use his time in Saint Louis as a valid training set. If you use the 2012-2014 Rams as your guide you can predict the 2018 Seahawks run vs play calls correctly just over 65% of the time. Not bad, but what about using the league? Basically the same result at 64%. While that's not bad, it may come as a surprise to you. Well, all three of you who made it this far into this fan post. Didn't we hear complaints about Schottenheimer being overly predictable? Especially with regards to the Dallas playoff game? And how can incorporate 2018 season information?
What if we use the 2018 regular season data only for Seattle and train the same type of model to predict the play calling in the playoff game versus Dallas only? And again, keep in mind this is a simple model using ONLY the information on seconds remaining in the half, down, yards to go, and score differential (field position was not predictive, but it was close!).
75%. Of the 54 offensive snaps counted in our data set (PFR says 55, slight discrepancy) our logistic model predicts Seattle's 2018 playoff game run:pass ratio with 75% accuracy.
One of the reasons our model fails to predict the 2018 Seahawks using Schottenheimer's old Rams data or league wide data is because they were in a class all to their own when calling run versus pass. Keep in mind the Bevell-hawks of 2017 were the second most situation pass happy team of 2017 in the regular season. Seattle in 2018? Completely different story.
The 2018 Seahawks weren't just run heavy when accounting for game state, they were the most run heavy team by a country mile. They were almost as run heavy as the Steelers were pass happy. Why is this significant? Well, in my mind there are a few obvious takeaways from this and it's getting late so I'll start wrapping up.
First, Seattle was only unpredictable if you looked at the league as a whole. If you took them at face value, i.e. when they said they were going to run even when people expected it. Well, that's exactly what they did. And not only that, accounting for game state let's compare them to the 2018 Patriots.
Seattle didn't just run more when others would also, they got more conservative in obvious passing downs. The farther from zero a team is in the above graph, the farther a play was from what the 20 nearest neighbors would do in the same situation. So, if we compare these two curves we can see that in obvious rushing downs Seattle was rushing more heavily than New England, and passing significantly less in obvious passing situations.
Of plays where Seattle was rushing when 75% of their 20 nearest neighbors wouldn't, the overwhelming majority are on early downs.
Of those same plays, again where the Seahawks were going against the grain of 75%+ of their league contemporaries, most of them occurred when the team was trailing, including the fourth quarter.
But did it work? After all, if Seattle is zigging while the league zags with great success then maybe there's nothing to be all that concerned with?
The evidence suggests no. Using win probability added for a particular play we can look to see if Seattle helped their chances during these times. Below is the distribution of outcomes for when the residuals exceed .75, i.e. when more than 75% of the nearest neighbors would not have run in a given situation.The solid bar representing the average outcome.
Additionally, while writing this over the last little bit an article stumbled it was across my feed. If you are so inclined, the author Jacob Davis looks at 2018 play calling deviating from expectation in a different way. He uses a logistic regression instead of KNN to set up what's "expected." He finds that while most teams benefit from deviating from league expectation, Seattle did not.
Wrapping it all up
Deviating from expected behavior isn't a bad thing, and if we look at league behavior in general or even his time with the Rams it appears that Schottenheimer is less predictable than Bevell. But, if we look at his Seattle play calling in isolation a more worrying outcome jumps out. Yes the Seahawks defied league trends, but they did so in a very consistent manner and if we use their regular season 2018 play calling we can identify Seattle's postseason play calls against Dallas correctly 75% of the time.
I didn't write all of this to dump all over Schottenheimer, or Pete Carroll, or the Seahawks. But before I stopped writing at Field Gulls for mostly personal reasons, I wrote in part about how Carroll's team appeared to be getting more risk averse over time and ignoring analytic trends. Now, maybe that's not what happened last year. Maybe the coaching staff just saw the run was working and rolled with it. But there is a very real possibility that someone at Dallas took a quick look at Seattle's play calling trends, saw these same results, and called their game plan accordingly.