Brandon Mebane, Pro Football Focus and a Circle of Handshakes

Three forces of human nature suspended and imbued this belief with detail: ignorance, hope/fear and plausibility. Venus was no longer just a pinprick of light or thought to be a nearby star, but its encompassing cloud cover prevented richer analysis, analysis which could penetrate the planet's atmosphere to the planet's surface. Having little definitive information, purveyors of hope/fear, both the speculative and actual, the non-fictional and fictional, envisioned Venus as something rather than nothing. That is, a place of richness, adventure and possibility, and not a lifeless rock entombed in a suffocating shell of sulfuric acid. Whenever a wise man says "we don't and can't know," an enterprising man will say "we will and do know, and it's awesome!" or "... it's terrifying!"

We now know, and I should say, we the average people now believe Venus to be a lifeless hellscape because experts have interpreted better, more reliable and more definitive information about the planet. We don't know say how we know what our dog's breath smells like or how to tie our shoes, but Venus as yet another uninhabitable sphere of rock and gas is accepted and well-supported, and fits our current understanding of space as vast, hostile and barren. Knowledge, ideas and belief, both galactic and ultra-trivial, change rapidly.

In a 12-year span from 1981 to 1992, three relief pitchers won the Cy Young and Most Valuable Player award in the same season. The cumulative wins above replacement of those three pitchers, 8.5, is two wins or about one Dustin Ackley short of Mike Trout's value in 2013, 10.5 WAR. Wins above replacement did not exist then and baseball writers inferred a value in the then new idea of a shutdown closer. Which value has since been proven not to exist.

Those writers were not stupid and certainly knew baseball. But what they wanted to see colored the truth, and the excitement and novelty of a new kind of hero, the shutdown closer, led to the kind of nonsensical and biased voting which in many ways fueled the modern sabermetric movement. Sports are ever infatuated with some new fling. The shift, the fly sweep, the 1080, the four-point play, PAR--sport is repetitive, especially on the macro level. Something new and well branded attracts readership and nowadays page views. If briefly, Jeremy Lin and "Linsanity" is every bit as valuable to the sports media as LeBron James was to the Miami Heat.

Modern statistical analysis of sport is, at its most popular, just another story to be exploited for as long as people find it interesting. More people have seen Moneyball than understand linear weights. And the Moneyball narrative, so simplified, is "out-of-the-box thinking allowed a small market franchise to compete in inherently unfair Major League Baseball." Which, simplified further, often reads as "stats are smart and modern," a shibboleth to distinguish the in-the-know from the old school and backward. This cultural telephone, relayed from iPhone 6 to rotary to tin can on a string, dignifies the statistical as smart however poorly conceived or executed the statistic.

Which brings us to the matter of Pro Football Focus.

A Madness to the Method

When Pro Football Focus began its website in 2007, it used a grading criteria one part scouting and one part algebra. I remember pass rushing grades looking something like this:

(Number of pressures x 0.5) + (number of quarterback hits x 0.75) ... etc.

With the value of each type of play being impossibly neat as above and not seemingly based on anything but simplicity and intuitiveness. That "pressure" unlike a home run or even a sack is nebulous seemed problematic. That fact-checking Pro Football Focus's data was impossible because of its decision to not relate data back to individual plays, but instead run it through an obfuscating and arbitrary formula seemed damning. The method stunk the duel stink of sloppiness and secrecy.

Yet now not ten years later Pro Football Focus is borderline mainstream. It has a recurring article at The Seattle Times. It "powers" the incomprehensible fantasy projection in my Yahoo! leagues. NFL coaches have singled it out for criticism. Its metrics are relevant. People I once worked alongside like Doug Farrar of Sports Illustrated cite and defend its data. Its numbers are used as credible shorthand when appraising a player's performance.

And before I get any further, allow me to acknowledge the nobleness of the cause. A site which breaks down publicly available game footage to assign individual grades to players is something of a Holy Grail in football analysis. Any smart football fan knows that individual statistics like passing or rushing yards do not represent one player but the play of the team, and yet everyone would love to know who is truly better Marshawn Lynch or LeSean McCoy, Russell Wilson or Andrew Luck, J.J. Watt or Robert Quinn. What I am writing is not meant as a defense of the opaqueness of coaches, or an attempt to evaluate Pro Football Focus against the impossibility of perfection. My intention is clear. I wish to expose and totally discredit Pro Football Focus.

Nowadays Pro Football Focus is even less transparent about their grading. This is how they describe the process:

3) How Do We Grade?

Each grade given is between +2 and -2, with 0.5 increments and an average of 0. A positive intervention in the game rates a positive grading and vice-versa. Very (very) few performances draw a +/-2 rating. In fact, the distribution of non-zero grades is like this:
.

+2.0	0.01percent
+1.5	0.3percent
+1.0	16percent
+0.5	37percent (unbalanced because of the way WRs and HBs are rated)
-0.5	24percent
-1.0	22percent
-1.5	0.5percent
-2.0	0.01percent

The grading takes into account many things and effectively brings "intelligence" to raw statistics.

That distribution of grades forms a bell curve because like much of the methodology of Pro Football Focus, there is an attempt at plausibility achieved by wearing the trappings of statistical analysis. In the very next paragraph, a whole set of conditionals is explored, but how those conditionals affect the grade are only hinted at.

For example, a raw stat might tell you a[n offensive] tackle conceded a sack. However, how long did he protect the QB for before he gave it up? Additionally, when did he give it up? If it was within the last two minutes on a potentially game-tying drive, it may be rather more important than when his team is running out the clock in a 30-point blowout.

An offensive tackle concedes a sack but PFF understands that not all sacks conceded are the same. The length of protection and the game state factor in. How and to what extent? That seems to be at the grader's discretion. Meaning, simply, that the number is a front for an opinion we can only guess at.

Like we can guess why a play worth -1 is 2,200 times more likely than a play worth -2. Excluding special teams, 32 teams playing a 16 game season, and 22 players a play, means Pro Football Focus claims to have awarded 58,007 -1 plays in 2013 but only 26 -2 plays. That's a bit like if for every double a batter hit, an average of 2,200 singles were hit in between. Or for every ten yard rush on first and ten, all runners ran for 2,200 five yard rushes on first and ten. And so forth.

There is no underlying rationale to this methodology. It is appears as a normal distribution in order to attain what Steven Colbert called "truthiness." Plausibility.

The About page marked "Grading" is full of this kind of equivocal, truthy and plausible rhetoric and encoding.

If you're not 95 percent sure what's gone on then don't grade the player for that play.

My brain lacks a percentage certainty meter.

We treat Ray Lewis as Baltimore No. 52 and see what grade he comes out with at the end for the individual performances in that game.

In other words, we know we are making subjective measurements of players we are predisposed to think good or bad of, but our subjective grading overcomes those biases because we want it to.

Many people say that as soon as you start grading, you bring subjectivity into your work. Obviously, to some degree, that's true.

Okay, here's a start ...

However, there's also subjectivity around whether a play was a QB run for negative yardage or a sack, if an assist on a tackle should be awarded and if a catch was dropped or not. Sure, you can come up with a set of rules to determine which is which, but in the end, at the borderline between one and the other, it's always subjective. It comes down to a judgment call.

Some very few official statistics are somewhat subjective too.

The real trick of grading is to define a clear set of rules, encompassing each type of play. If your rules are thorough and precise enough, the answers just fall out. It becomes as easy as determining the dropped pass that hit the TE right between the numbers.

Never you mind what those rules are, they are "thorough and precise." But!

Just like with the more mainstream statistics, there are occasions when the choice is difficult. But the difference on our site is this: If a guy is going to be upgraded or downgraded on a judgment call, we let it ride. We simply make the comment and then put in a 0.

The difference at PFF is that when they don't know, they simply punt responsibility.

Statistics in their raw form are considered objective. But in our opinion, with the small number of NFL games played each season, raw stats are very often unintelligent. If a QB throws three interceptions in a game but one came from a dropped pass, another from a WR running a poor route and a third on a Hail Mary at the end of the half, it skews his stats by far too great an amount to be useful.  Our "subjective" grading allows us to bring some intelligence to the raw numbers.

Having not explained away the inherent subjectivity of their grading system and the massively undermining implications of that inherent subjectivity, a straw man is hoisted to distract and attract scorn. Sometimes--maybe at least and however likely--interception totals, which measure something with certainty but which can have several causes, "skew stats." But we are not told which stats or how they are skewed. Finally, as if problem overcome, the word subjective is put in scare quotes, as if calling PFF's statistics "subjective" is an unfair bit of slander, and we are reassured the grading "bring[s] some intelligence to the raw numbers."

It does not. It is a step backward. No one is likely to read content generated by Pro Football Focus and become more informed. A bad, slippery method conducted behind closed doors produces nonsense like this:

Defensive tackle Brandon Mebane with another bad game, making it four "red grade" games out of seven on the year. His -3.4 this week was third-worst for DTs playing more than 25% of the snaps. Unfortunately, his -3.8 run defense grade was the worst in the same category.

And, at the very least, it is time for anyone who wishes to be taken seriously as a sports journalist to stop taking seriously the work of Pro Football Focus.

Brandon Mebane and an Attempt to Tip the Scales

Expectation and Performance

Before I get into the snap by snap analysis of Brandon Mebane (I did every snap but will isolate a few for analysis), here's a pretty basic question that should probably be answered. How does a metric composed of subjective awarding or debiting of "0.5 increments" arrive at a score of -3.8? Let alone that I am truly not sure what -3.8 means other than "bad," the number seems impossible to arrive at with the process PFF lays out as its own. A -3.8 suggest a far simpler method of evaluation doesn't it? That is: watch game, and turn general impression of good or bad into a number like a film or music reviewer might. But I digress.

Longtime readers of Field Gulls know I am a fan of Brandon Mebane but why I will explain a little further down. It is probable I favor him and overlook some of his shortcomings. His is unlike any other player's assignment. Mebane must on snap after snap do his very best to survive an inherently disadvantageous matchup--in fact two. He is the team's designated dampener, a player who attempts to break even against near every snap double teams. This he does as well as anyone I know. In the words of defensive coordinator Dan Quinn:

One of the things he has for being a big guy is his balance. So when you get two guys on you, a lot of times somebody can push on you on your hip and you get movement and you can be on the ground. So that's one of the things that's so remarkable for a 330-pound guy, that he's not on the ground and he has great leverage. He's built that way, but he uses it to his advantage. For his size, the thing that we always admire about him, we rarely see him on the ground. It's the leverage that he plays with that he can get underneath the blocker.

Richard Sherman calls him "the immovable object."

No one who knows anything about football expects Mebane to regularly beat a double team. It is enough that he slow or stymie the movement of both linemen, and/or keeps both engaged as long as possible, because, as Bobby Wagner says "[Mebane's] the reason I play as good as I play. He does a great job keeping guys off of me."

This, which may appear to be a lost matchup ...

Allows this ...

Which leads to a surely successful play by the defense.

Which shows that Mebane may only be graded if you have a sound and consistent criteria for grading him against his peers. Otherwise apples are being compared to oranges. Mebane did not force back the Carolina offensive linemen, he did not split them and make the tackle himself, but against two players working in concert to force him back and maybe knock him down, he stayed up, stayed close to the line of scrimmage, and prevented pulling linemen to block out the linebackers.

On most snaps, that is Brandon Mebane's job, and on most snaps, that is what he does.

Not always ...

But even when he's badly losing to a double team, Mebane is typically good enough to prevent the second blocker from pulling free and engaging a second-level defender.

Nor is Mebane double-teamed strictly because of convention. He is rarely single-blocked on runs plays, and when he is, it's typically obvious why such an occasion is rare.

Here Mebane is singled against Ryan Kalil. Jonathan Stewart is attempting to run right.

By single blocking Mebane, the left and right guard are freed to pull into the second level. But where a hole should be, there is instead Kalil and Mebane flowing right and walling off the rusher.

Stewart moves further and further right, never truly able to find a hole, and when he turns up field Tharold Simon is there to tackle him for no gain.

I might evaluate those plays at 0.5, -0.5 and 0.5 if I were grading by PFF's supposed method. But even typing that makes me feel foolish. Analysis means to break into smaller parts. So, to analyze a game, we might analyze the individual performances of players play by play. Which takes the raw tape data and expands it.

Abstracting a number from one player's performance functions in the opposite fashion. It takes a collection of data and contracts it. Information is lost, nearly all of it, and the resulting number, the grade, teaches us nothing, allows for no discussion, and virtually ensures corruption in the form of bias, sloppiness and a lack of accountability.

You may agree or disagree with how good Mebane performed in each of the above plays. You may revisit the game and watch those plays yourself. The process is open.

Competition Neglect

Overall I do not think of last Sunday's game as a great game by Mebane. He mostly worked to a stalemate against near-constant double teams. He has done better. But Kalil is a four-time Pro Bowl center, and he's the second highest paid center in the NFL.

Which highlights another fatal failing of PFF: competition neglect. Pro Football Focus should be uniquely able to handle the difficulty of assessing individual performance against competition, but the analysis surrounding their numbers is typically so shallow, one can't even be sure who a player played against. Instead we get something like this:

Signature Play: 3Q, 6:18 – Used a spin move effectively to get inside and land a hit on Smith.

Which is almost identical to that which is contained in the play-by-play.

2-8-DEN 8 (6:18) (Shotgun) 11-A.Smith pass incomplete short right to 87-T.Kelce [94-D.Ware].

Who was blocking Ware? Was he chipped? How long did it take for him to arrive? Did his pressure have any part in Smith's incomplete pass? This was DeMarcus Ware's "Signature Play" and the information Pro Football Focus provides beyond the basics found in the NFL's play-by-play, is that Ware used a spin move.

Quality of competition does not get factored out in the NFL the way it does in baseball and basketball. Teams only play 16 games, and because of an unbalanced schedule which pits division rival against division rival twice each year, some teams, like the 2014 New Orleans Saints, have faced opponents whose combined point differential is -88 (2014 Tampa Bay Bucs -90). While other teams, like the Indianapolis Colts, have faced opponents whose combined point differential is +75 (2014 Denver Broncos +82.)

Quality of competition is essential to understanding if an individual player is doing well or not. Without it, you are left with little more information than be found in a box score. Is being beat on a jump ball by Calvin Johnson the same as being beat on a jump ball by Donnie Avery? What if Darryl Tapp beats Russell Okung around left end? Is that equal to if Justin Houston does likewise? Of course not. Football is a matchup sport. Pro Football Focus attempts to evaluate performance while ignoring who that performance is against.

The Confusion of Awarding Individual Credit for Team Play

Brandon Mebane's final snap ended with him sacking Cam Newton, which play improved Seattle's chances of winning from 28% to 43%. It was a good play by Mebane. One not unlike many good plays he made in passing downs, in which he forced his blocker back but never really neared Newton enough to separate and attempt a sack. What differentiated this from those plays was the performance of his teammates.

Seahawks set in a 3-4 look. Both ends are over both guards, and Mebane is playing nose/1-tech right.

Seattle was able to stuff the run on the previous play (the play described above in which Mebane survives a double team and Wright closes to tackle Stewart at the line of scrimmage) and are anticipating a pass by Carolina.

Carolina does pass. A play action is aborted and rookie running back Darrin Reaves, since waived, fans out to block Bruce Irving coming around left end. All together, Carolina has seven blockers and only three receivers. Coverage is mostly solid, though this is aided by the fact that few routes have concluded by the time the Seahawks pass rush has begun to arrive.

Sherman is a little loose on the right and may have been caught looking at the quarterback, but Sherman is notorious for turning off coverage into an interception. Whatever the case, Newton is too harried by rush to see Benjamin.

Kalil's line read has Carolina's blockers sliding right, which means undrafted rookie and replacement left guard Andrew Norwell is singled up on Mebane. Cliff Avril around right end flushes Newton forward, Greg Scruggs flushes Newton right, and as Newton attempts to escape right, Mebane disengages from Norwell and sacks.

Good play call by Quinn. Good execution by Scruggs, Avril and Irvin. Good coverage by Seattle's linebackers and secondary, or so it seems. And a very easy matchup for Mebane, but he converts and earns the sack.

Would you rate Mebane on this play a +1, a -0.5, a +0.5, a -1.5 ...

In Summation

Pro Football Focus grades players snap by snap. They claim to account for game situation. They also claim to account for opposing player and matchup, but that information is rarely included in their analysis. All of this supposed richness of information is reduced to a nine point scale (which, according to them is really a seven point scale, since +2 and -2 plays happen about 50 times a season.) And that grade is determined by one person, who does not explain how he arrived at that grade or what it means.

Analyzing one player and his grade for one game was more than sufficient to prove how hopelessly flawed this method is. No standard is given for how a player should perform against a double team. No adjustment for quality of opponent is accounted for. And, after watching every snap played by Mebane multiple times and in slow motion, I cannot even guess how Mebane's -3.8 against the run was reached. Somehow Mebane rated as the worst regular defensive tackle in the NFL in week 8, when despite injured/out-of-position/rookie linebackers and a hobbled strong safety, the Panthers averaged 3.3 yards per attempt on 17 rushes between left guard, center and right guard. That same week, the Texans rushed for 149 yards and two touchdowns in 20 rushes at the Titans left guard, center and right guard.

The Internet and the Circle of Handshakes

No one seems to be as on to the ridiculousness of Pro Football Focus as their commenters. Nearly every comment section I ran into while researching this had a comment much like the one below.

OrPickAName • 5 days ago

-0.2 for Foster? I'm beyond trying to understand how you grade running backs.

That comment came after Arian Foster ran for 151 yards and two touchdowns in 20 attempts. It is of course possible that Foster was so aided by his blocking that he really did perform below average as a rusher despite his overwhelming success, but it's unlikely. And it's unlikely grade after grade, if accurate, would confuse common football fans.

Innovative stats may highlight lesser appreciated but good players, or expose flashy, high-profile players who, perhaps, hurt their team by taking plays off or missing assignments or dropping passes. Given millions of eyeballs watching NFL football every week, and countless amateur scouts working online, one can't help but wonder how secret of a superstar Tyler Polumbus really is. Polumbus was benched after allowing nine sacks in seven games.

But if Pro Football Focus is respected by few, relies on a deeply flawed method, hides its data behind arbitrary figures, and seems to do nothing to enrich our appreciation of the game of football, why is it so relevant? Truth be told I can not answer for sure but I will offer a guess or two.

A Culture Against "Hating"

The largest age group of Americans, of which I belong, is the Millennials. Millennials are young, aspiring, have been coached to dream big by their parents and by their culture, and perhaps because of those big dreams, do not like to "hate" on others. Excusing the irrelevant and anonymous internet commenter, those young people aspiring to be sports journalists, and there are thousands, realize that little is gained by making enemies.

Success today is about networking and on a massive level. Pro Football Focus has 115,000 Twitter followers. It is a better friend than not-a-friend, and lots of respectable people follow its Twitter feed.

More to the point: Once someone or an entity builds sufficient popularity, one cannot ignore them without hurting one's own relevance. No one at Football Outsiders seems to respect the work of Peter King, but his column is linked to every week. That linking is likely reciprocal and though Football Outsiders does good work and King is an old hack and a sinecure, King helps stave off deserved irrelevance by tapping into Football Outsiders' credibility and Football Outsiders draws mainstream exposure by being linked to by King.

This is what I call "The Circle of Handshakes." Twenty years ago, independent musicians legitimately stressed about "selling out," and top Hollywood actors wouldn't be caught dead in a stateside commercial. Now Samuel L. Jackson barks at us about Capital One and being featured in an Apple ad launches young musicians' careers. Artistic credibility is laughed off. What matters is how many monied and powerful parties you can recruit to invest in your success.

Pro Football Focus wields the power of internet relevance. And where there is internet relevance, money often follows. Thousands are striving toward a dream few will achieve. People want an "in," connections, and that means networking, and networking means empty, business-like relationships built not of mutual respect but of mutual self-interest.

Free Content

People expect a certain quality from just about anything they pay for, but will do almost anything provided its easy and free. The internet is the definition of easy and free, and has thus almost no quality control. Further, the model for greatest success is maximum amount of content; quality be damned. Half of the stories which appear on the front page of ESPN.com are deceptively worded headlines about something trivial. This is called "click bait" but find me the successful site which doesn't participate.

Pro Football Focus not only provides free content for its readers, it provides free content for blog writers. Even if it's just a copy and paste and a "boy these guys are schmucks," it works. It works for Pro Football Focus, which spreads its influence, and it works for the blog writer, who maybe worked no more than 20 minutes on said post.

Simplicity and Plausibility

So complex is player evaluation in the NFL, even reading a detailed analysis of a player takes more time than most care to invest. Pro Football Focus understands that people don't want depth and complexity. They want a simple, easy answer with the whiff of sophistication. Reducing a supposedly comprehensive analysis of a player into a number does just that. And, surely by design, that number looks like real metrics such as WAR and EPA and WP48.

Pro Football Focus isn't perceptive, intellectual or analytical but neither are most sports fans. Its simple numbers pablum is more than enough when conversation revolves around who's overrated and who sucks.

People Have Accepted the Process and so the Results Are Academic

How hard is it to determine J.J. Watt is good or Geno Smith is bad? Once people have gotten over that the process makes no sense, it's easy enough to fudge numbers which seem close enough to truth to seem substantial. This is one of those times when the process is definitive. If we do not know it fully, but what we knows stinks, the results are immaterial. In fact, given the nature of PFF's process, that the results seem accurate enough is the only retroactive credibility it has. Which means, basically, that so long as its lies confirm our assumptions, we will assume those lies to be truth.

In Conclusion

Like so much of the modern world, Pro Football Focus seems harmless but is in fact pernicious. It does not steal your money. It steals your time. It does not lie to you directly. It mystifies with numbers and hides the truth beneath layers of confusion. It is not innovative or informative. It is a new and fashionable way to con goodhearted people who have, maybe, not been burned by con-men or who don't care or who simply wish to con the con, so long as the latter con is victimizing someone else.

Pro Football Focus is a tiny, tiny thing and not at all likely to destroy the world. I hope people read irony into that. But its success is something else entirely. It relies on that same admixture of plausibility, cognitive ease and inborn biases which prop up such dangerous nonsense as conspiracy theories, pseudoscience and obscurantism. We live in a time warped by plausibility. Any two-bit theory can be supported and every consensus scientific opinion can be undermined.

But this is about football. Pro Football Focus is a secretive, inconsistent, nonsensical and pretentious method of evaluating football players. It is very much anti-journalism, misinformation, a noble attempt botched and covered up. Until it embraces transparency, consistency of method and drops the arbitrary number values, it will remain a very successful, very popular and totally disreputable website. Something which siphons from the popularity of the NFL and offers nothing in return.

As for Brandon Mebane: When I started writing about football, I was a fan only, and less informed than many. My ambitions greatly surpassed my ability or expertise. Mebane was a mid-round draft pick in a defensive tackle rich class. He was anything but flashy--could have even been called undersized at a position which depends so much on size. I remember very distinctly just how critical I was of his drafting. I had no access to his tape from college, and little of the knowledge needed to evaluate that tape. But I had a VCR, yes a VCR, and the preseason is free for us diehards who wish to watch it.

I am an intense and hopeless autodidact. Understanding why Mebane was good and defending that position with publicly available information, taught me not only so much about this game I love, football, but how to learn, how to think and how to know. His career has been no less than a great education. He, to me, will always be football and fair competition at its purest.