The Hopeful Side of Harsh Criticism

by John Morgan on Jul 30, 2010 4:09 PM PDT

Nate linked to Brian Burke's analysis on season projections down below, but here's the link and here's the conclusion:

This is a notable result because it suggests that Football Outsiders’ predictions are so bad, they are literally worse than having no football knowledge at all. It's like negative information, draining us of insight.

Oof. Brian then invokes Billy Madison, and this scene.

I have a lot of respect for some of the work Football Outsiders does and many of the people that work for Football Outsiders, but it's information like this that made me wary of much of their analysis. Bad information does indeed make someone, if not stupider, more capable of stupid ideas and decisions.

However, I do not think the story here is how inaccurate Football Outsiders' projections are, but rather the still impenetrable world of the NFL and how very difficult it is to predict the NFL season. In Major League Baseball, we can measure the speed and arc of a pitch. We can create similarity scores from over a hundred years of history. Well not we, but smart people that do that sort of thing. Baseball analysts can even determine the raw value of a single versus a double versus a pop fly.

Not only is it difficult to measure the exact value of any one football event, it's impossible to isolate the contributions of one player among the 11.

This is all ground I have covered before, so I will save you the long, philosophical blahblahblah. I do not come here to bury analysis and prognostication but to speak of hope. The hopeful side of Burke's scathing critique is that even with thousands of hours committed from some of the finest football minds, the ultimate outcome of the NFL season is still impossible to predict.

And that means, however marginal, even if I say it's not realistic, even if any rational person without an agenda says it's not realistic, the Seahawks could, maybe, could, maybe maybe maybe, not finish in last place. We still play the Rams, you know?

Tweet Comment 10 comments | 0 recs |

Do you like this story?

Comments

Slight correction...

We still play the Rams… TWICE!

by zeeehjee on Jul 30, 2010 4:14 PM PDT reply actions

Little news there

I love the work of FO when it comes to analyzing games and playbooks.

I do not think statistical analysis has a very high predictive value. Not in any sports, but particularly not in football (both sports).

FOA is a good read because it goes blow-by-blow for a team’s make-up and staff, strengths and weaknesses. It’s a good read on teams I don’t know that well. The predictions are fun but practically useless.

by Thomas Beekers on Jul 30, 2010 4:31 PM PDT reply actions

Agreed, the value to me is not in prediction, but insight.

Still, I think Brian is being a bit harsh. Single-sampling is a pretty thin platform to levy criticism from. Two years. 2007, and then 2009, including the highlight of the Jets. Wouldn’t that be anathema to him?

At least 6 years, I think, to start ought to be in scope. But years, and number of teams, in scope doesn’t mitigate the variance in value of each hit/miss. Maybe after a dozen, it would. But hitting on a middling team like the Bengals or Texans, to repeat as middling, is not the same coup as being among the first to hit on semi-longtime bottom feeders San Diego to rise to perennial contenders. Or the Seahawks. Or to hit on preseason conference favorites Dallas to stumble, a greater coup than to hit on the Raiders to suck again.

Measuring that value is arbitrary. I wouldn’t suggest trying to quantify it. But the difference in value of particular hits and misses is clear and apparent, to me.

Burke just has too much of a vested interest in criticizing FO, to not take it with a slight grain of salt, for me. He’s targeted them at least a few other times. It’s kind of like listening to Steve Balmer tell me what’s wrong with the iPad. I respect his methodology and findings more than theirs, but that is not to say some of their work isn’t pretty good. And predictability is not a comprehensive measure of whether their work is good, even if they are using it to predict and sell the predictions.

by jacobstevens on Jul 30, 2010 4:59 PM PDT up reply actions

I forgot to add

that missing on the Rams, to not suck, is definitely more egregious than missing on the Seahawks to not suck. I believe the FO predictions are polished with some subjective tweaking, common sense or otherwise, and I wonder if they got too cute with St. Louis, zealous to hit on a team no one else would put money on.

They also seem to me to put too much weight and confidence in regression to the mean. It’s one thing to recognize the potential for it, and another thing to bank on it. That’s gambler’s fallacy.

by jacobstevens on Jul 30, 2010 5:06 PM PDT up reply actions

Holy economic statistics Batman!

Can I get a two tailed T test with that ANOVA?

All The Way, AIRBORNE!!!

by Airborne Hawk Guy on Jul 30, 2010 5:20 PM PDT up reply actions

I miss D.J. Hackett.

Irrational love is a dangerous thing.

by Hopefulmsfan on Jul 30, 2010 8:21 PM PDT reply actions

That's ok.

Deion Branch is the new D.J. Hackett.

Red Bryant: surprise us!

by Misfit74 on Jul 30, 2010 10:03 PM PDT up reply actions

Lies.

Hackett was the first Seahawks receiver I developed a strong liking for.

Though they sink through the Sea, they shall rise again...Death shall have no dominion...

by Cheddar28 on Jul 31, 2010 3:03 PM PDT up reply actions

And you had to love his celebration he does with his hands.

by Seahawksfan23 on Jul 31, 2010 5:05 PM PDT up reply actions

One year!?

Are you kidding me?

I don’t know how any one could possibly undermine their credibility in statistics any more than Burke just did. Dude didn’t even MENTION the fact we need to back off the conclusions a bit due to the absurdly small sample size. Five years should be just about the minimum anyone should even dream of comparing and realistically it should be more like 15+ years. Remember, the fact that FO’s methods were developed more recently is irrelevant, techniques can be backtested against historical data. The fact that FO developed the methods from backtesting would make it pretty easy for their predictions to fare well, but this is sort of the problem with statistical methods in general.

The conclusion that FOs’ methods aren’t iron-clad is a fair one (and obvious), but that they are actually not as good as the null or simple rule-of-thumb predictions is a big overstatement.

This is pretty benign because we are just talking football here, but the misrepresentation of data or use of just plain bad method is pervasive in the blogosphere. This gets pretty dangerous when it is done for health, environment, etc. issues and people actually form their views based on armchair bloggers.

John Morgan is unassuming, has integrity, and knows that doing the research is what gives you the right to speculate conclusions and that is why I follow his blog.

by michaelfox99 on Aug 1, 2010 3:13 PM PDT reply actions