NFL Playoff expansion is objectively wrong, part II

In Part I, we dismissed some of the lame arguments in favor of playoff expansion. We concluded with a simulation that showed a 6-team (per conference) playoff format is better than a 5- or 7-team format, across the board, at producing a Super Bowl or Conference Championship win for elite teams. Part II will analyze the impact on regular season games.

Quantifying Harm to the "Regular" Season

The thing that's great about NFL football-- for now-- is that every game counts. The NBA Champion San Antonio Spurs had a three-game margin over the next-best team in their conference. A fan who missed several of their regular-season games wouldn't have missed anything of impact; for that matter, even earning the top seed was not important, because there is no bye week and each elimination round involves a best-of-seven series (and consequentially, 1/7th the home field advantage of an NFL playoff game).

By contrast, the Seattle Seahawks and Denver Broncos each needed every single win to secure a #1 seed, and the Seahawks were one game away from not even getting a bye.

To measure the impact of winning in the regular season, I reprogrammed the simulation to halt after 14 games. It then "marked" for consideration each contender (defined as a team with 8 or more wins at that point) and ran a sub-loop 100 times to simulate just the last two games and the playoffs.

Thus, the program could answer the question of how much an 0-2 finish, a 1-1 finish, or a 2-0 finish contributed to a contender's chance of making the playoffs or a winning a championship.

Each win bucket was weighted equally. This is really, really important. If the league's best team went 2-0 in 90 out of 100 sub-loops, winning 45 Super Bowls, that was counted as a 50% Super Bowl Championship rate for the 2-0 bucket; if the league's 9th-best team went 2-0 50 times out of 100 sub-loops, winning 10 Super Bowls, that was counted as a 20% Super Bowl Championship rate for the 2-0 bucket.

In other words, the sub-loop does not measure correlation between winning in the regular season and winning in the post-season. It only measures causation (the direct impact of earning a playoff spot or particular seed).

(In the simulation, these two games were always non-divisional conference games to provide "average" impact as compared to division or non-conference games)

Results: Exactly what I expected:

When there were seven teams admitted, the promised excitement of playoff races did not materialize. Quite the opposite. With so many playoff spots available, a 2-0 finish increased a team's relative probability of getting into the post-season by just +44.6%, compared to +73.2% in the current format.

("Relative probability" means that if you go from 20% to 30%, it's a +50% increase; in this case, going from 68.64% to 99.28% is a relative increase of +44.6%)

If five teams are admitted per conference, winning became even more important for a single team's relative probability (going 2-0 improved your chances by +86.7%), but had less absolute impact than it did in a 6-team format. Two wins earned 3962 additional playoff berths with 5-team format and 4049 additional playoff berths with a 6-team format.

	achievement:	Made Playoffs			Conference Champion			Super Bowl Champion
FiveTeams per Conference	last two games:	2 wins	1 win	0 wins	2 wins	1 win	0 wins	2 wins	1 win	0 wins
	Per 10000	8509	6568	4557	1795	1291	933	878	637	484
	increase/ 1 win	1976			431			197
	pct increase/ 1 win	36.6%			38.7%			34.7%
	increase/ 2 wins	3952			862			394
	pct increase/ 2 wins	86.7%			92.4%			81.4%
SixTeams per Conference	last two games:	2 wins	1 win	0 wins	2 wins	1 win	0 wins	2 wins	1 win	0 wins
	Per 10000	9577	7940	5528	1754	1283	838	848	633	416
	increase/ 1 win	2024.5			458			216
	pct increase/ 1 win	31.6%			44.7%			42.8%
	increase/ 2 wins	4049			916			432
	pct increase/ 2 wins	73.2%			109.3%			103.8%
SevenTeams per Conference	last two games:	2 wins	1 win	0 wins	2 wins	1 win	0 wins	2 wins	1 win	0 wins
	Per 10000	9928	9090	6864	1680	1288	916	816	631	450
	increase/ 1 win	1532			382			183
	pct increase/ 1 win	20.3%			35.4%			34.7%
	increase/ 2 wins	3064			764			366
	pct increase/ 2 wins	44.6%			83.4%			81.3%

The current system trumped both others in terms of Championship impact. Going 2-0 over a two-game span more than doubles a team's chance of winning a Conference Championship or Super Bowl.

It's the bye week, stupid.

Regular-season games as Playoff Equivalents

Let's start with some hypotheticals to get an intuitive grasp of this part. Imagine a 3-game regular season where you have to go 3-0 in order to make the playoffs. If you look at one game in isolation, there is a 75% chance that it's meaningless because you lost (or will lose) at least one other game. There is a 25% chance that you won (or will win) both of the other games, in which case the third is essentially a playoff game where you will be eliminated or else advance. Ergo, an average regular season game is worth 25% of a playoff game when measuring advancement impact. The aggregate value of the all three regular season games is .25 + .25 + .25 = .75 playoff games.

Now lower the bar on our three-game season and require just two wins for a playoff berth. If you look at one game in isolation, there is a 25% chance that it's meaningless because you lost (or will lose) both others and are eliminated regardless; and there is a 25% chance that it's meaningless because you won (or will win) both others and advance regardless. Ergo, an average regular season game is worth 50% of a playoff game when measuring impact on advancement. The aggregate value of all three regular season games is .5 + .5 + .5 = 1.5 playoff games.

This may seem counter-intuitive, because each regular season on the whole results in elimination or advancement. Partly, that's because we aren't accounting for the fact that a playoff berth is more exclusive (and therefore more valuable) when you have to go 3-0. Try to ignore that for the moment and think about the relative excitement of each game. With the 3-win requirement, you can never advance by winning the first or second game, and a large chunk of your later games will be meaningless (because you're already eliminated). With a 2-win requirement, the first game has a big impact on your future chances; the second game always presents a chance to be eliminated or to else a chance to advance; and the third game is effectively a playoff game half the time.

So the hypothetical 2-win scenario has more playoff-related impact during the regular season than does the hypothetical 3-win scenario. Objectively and measurably.

It's impossible to draw up a complete probability tree like that for a 16-game NFL season with a variable playoff threshold, so we use the brute force of a simulation. Across the entire league, an average regular season game is worth about 13% of a playoff game regardless of format. This is not shown on the chart because (for the moment) we are ignoring the wasted games played by bad and mediocre teams to focus on the impact for teams that are over .500. Among that selection, a regular season win adds 2024.5 playoff berths per 10,000 seasons, as shown in the middle of the left-hand column above. Ergo, a regular season game in the current format is worth 20.245% of a playoff game. And the entire season for such a team has an aggregate value of .20245 X 16 = 3.2392 playoff games.

Similarly, we can calculate an objective and measurable championship equivalent from the same simulation. If the 6-team playoff involved random seeding, random byes, and neutral locations, then a playoff berth would be worth exactly 1/6th of a Conference Championship, and we could simply divide the regular season playoff equivalent by six (20.245%/6 = 3.374%). But winning can do more than get you into the playoffs, it can give you home field advantage and/or a bye week. So the Conference Championship Equivalent is expected to be higher.

Results: Even more dramatic than I expected:

Focus on the boldfaced numbers at the bottom right. Goodell's proposal superficially adds the excitement of two playoff games. But in fact, among winning teams, it degrades the regular season by the equivalent of more than eight playoff games. If that number seems ridiculously high, consider that exactly eight playoff teams in 2013 finished at least a full game ahead of the #6 Wild Card in their respective conference; if there had been a #7 seed Wild Card slot available, there would have been 8 more "surplus wins" in the league that did not impact whether these teams made the playoffs (this is not precisely what the simulation measures, but it should help with perspective).

	equivalent to	value for one game	value for full season 16 games	Times about 14 Qualified Teams	Less Overlap (playing each other)
5-team format regular season win value	playoff win	0.1976	3.1616	44.26	34.98
	Conference Championship	0.0431	0.6896	9.65	7.63
	Super Bowl Win	0.0197	0.3152	4.41	3.49
6-team format regular season win value	playoff win	0.20245	3.2392	45.35	35.84
	Conference Championship	0.0458	0.7328	10.26	8.11
	Super Bowl Win	0.0216	0.3456	4.84	3.82
7-team format regular season win value	playoff win	0.1532	2.4512	34.32	27.12
	Conference Championship	0.0382	0.6112	8.56	6.76
	Super Bowl Win	0.0183	0.2928	4.10	3.24
Difference between six and seven	playoff win	0.04925	0.788	11.03	8.72
	Conference Championship	0.0076	0.1216	1.70	1.35
	Super Bowl Win	0.0033	0.0528	0.74	0.58

More alarming to my mind is the degradation of regular-season impact on the Super Bowl and Conference Championship. Is it really worth adding two low-grade playoff games at the cost of removing more than one Conference Championship and half a Super Bowl? Also, take note of the proportional difference between the Conference Championship impact and the Super Bowl impact. We might expect the latter to be exactly half, but it is less than half. What's happening is that part of the degradation in win value is "made up" for by the fact that, although good teams make the Super Bowl more rarely, they are more likely to face an inadequate opponent thanks to the same effect happening in the opposite conference.

Tweaking the selection parameter

You may be wondering why I used a cut-off that examined only winning teams in measuring relative playoff impact. There are several reasons.

Firstly, the value of winning a playoff berth to a 12-team tournament should actually be considered equivalent to 1.33 playoff wins, because 50% of teams advance in a playoff round and only 37.5% advance from the regular season. A playoff berth in a 14-team tournament would be equivalent to just 1.14 playoff wins, which is considerably less.

Secondly, and most importantly: The entire point of the playoffs is to determine Champions. This end result is more important than the prestige of making the playoffs or the perceived quality of teams involved, because it is what gives meaning to the first part of the process. So even if the 7-team format produces more equivalent playoff value in the regular season games, if it does so without producing more equivalent Championship value (both Conference and Super Bowl) we should be alarmed and in opposition. An increase in playoff impact without a corresponding increase in Championship impact is a red flag. It indicates that the Championship impact is still being degraded, it is merely disguised in the data by the increased playoff impact.

At any rate, here are the simulation numbers using a 7-win cutoff (counting teams at .500 after 14 games) instead of an 8-win cutoff:

	achievement:	Made Playoffs		Conference Champion		Super Bowl Champion
SixTeams per Conference	last two games:	2 wins	1 win	2 wins	1 win	2 wins	1 win
	Per 10000	8390	6282	1383	978	676	485
	increase for 2 wins vs 1 win	2108		405		191
SevenTeams per Conference	last two games:	2 wins	1 win	2 wins	1 win	2 wins	1 win
	Per 10000	9301	7398	1337	980	655	488
	increase for 2 wins vs 1 win	1904		357		167

And here are cumulative impacts that result:

	equivalent to	value for one game	value for full season 16 games	Times about 18 Qualified Teams	Less Overlap (playing each other)
6-team format regular season win value	playoff win	0.2108	3.373	60.72	44.07
	Conference Championship	0.0405	0.648	11.67	8.47
	Super Bowl Win	0.0191	0.306	5.50	3.99
7-team format regular season win value	playoff win	0.1904	3.046	54.82	39.79
	Conference Championship	0.0357	0.570	10.27	7.45
	Super Bowl Win	0.0167	0.267	4.80	3.49
Difference between six and seven	playoff win	0.0102	0.327	5.89	4.28
	Conference Championship	0.0024	0.078	1.40	1.02
	Super Bowl Win	0.0012	0.039	0.70	0.51

The damage done to playoff impact by a 7-team format goes down quite a bit, which is no surprise, because a number of 7-7 teams are now counted as having playoff-critical games. But we are still losing the equivalent of one Conference Championship and half a Super Bowl.

If you keep lowering the standards for which teams are counted, you can make the reduction in equivalent playoff impact (but never the reduction in Championship impact) disappear altogether. But having seen the numbers with a "contender" standard, here is my third reason for using it: We can now see exactly what happens with a 7-team format, which is more playoff impact for teams in the range of 6-9 or 7-8 at the expense of less playoff impact for teams that are 8-7 or 9-6.

Some other opinions, simulations

Neil Paine, writing for FiveThirtyEight, opines that a 14-team playoff would not significantly reduce the chance of the objectively best team winning the Super Bowl.

First, props to Neil for putting Richard Sherman's pic at the top of his article, and props for using the phrase "optimal mix of determinism and randomness."

I'd like to give credit for his use of a simulation, but he put the results in a low-resolution graph that makes it impossible to see his actual numbers. Moreover, Neil's narrow focus on the Super Bowl overlooks the point I've made several times already: The reduced likelihood of a top-four team reaching the Super Bowl is slightly mitigated by the fact that both Conferences are producing worse Conference Champions. An elite team will make it to the Super Bowl less often, but have a slightly better chance of facing a non-elite opponent.

This is not a good thing.

More fodder from Pompei via Bleacher Report, who claims "Another playoff spot in each conference would mean that more teams would be in contention late in the year, and there would be fewer meaningless Week 17 games."

Point of fact: It only takes three meaningful games to fill the week 17 broadcast schedule.

Also from Pompei: "It really does not matter if the NFLPA or a few old-guard owners oppose it; they can get in the way of more playoff games no more than bison herds could get in the way of railroad construction in the nineteenth century."

Conservation and biodiversity aside, this is a flawed metaphor. Railroads are an objective improvement over wagons for their added speed and capacity, and can therefore be replaced by successively faster and better modes of transportation, such as motor vehicles, maglev rails, teleportation pads, etc. If this were true of playoff expansion, then we can and should have a 32-team playoff bracket in place already. A better comparison would be an aesthetic form of entertainment that has an optimum (not maximum) measurable. Movies, for example, are not getting progressively longer.

And this: "The playoffs would not be diluted by adding two teams in most seasons. The 2005 Steelers and the 2010 Packers won the Super Bowl as No. 6 seeds. Why couldn't a seventh seed?"

A blatant slippery slope fallacy. The #7 seed could win, as could a #8 or #9 or even a #32 seed. But each successive team added is less likely to win and more likely to suck. Most years feature a Wild Card team getting blown out by more than 20 points. That's the cost we're already paying for the rare underdog like Pittsburgh or Green Bay.

From Mile High Report, user Topher Doll has an interesting idea for preserving some value in the #2 seed:

"The Twist: The 1st overall seed can choose home field advantage through the entire playoffs OR choose the first round bye."

I have no analysis of this, I just think it's brilliant and wanted to share.

Finally, Doug from Pro-Football Reference ran a series of simulations in 2006 to see how often the best, second best, etc. team won the Super Bowl. He did not simulate a 7-team format, but he simulated a number of others. Before running the simulation, he made the case for a 6-team format very eloquently:

"If you eliminate the wildcard, you will too often shut the best team out of the playoffs altogether.... And if you have more than two wildcards, you will too often make the best team navigate an extra round of playoffs."

But when seeing his simulated data, he was "floored by how little the playoff format seems to matter."

Kudos to Doug for quality work across the board. However, the limited scope of his final data and outdated standards for estimating quality distribution make the above conclusion wrong.

For one thing, he only measured the impact on winning the Super Bowl, and not the impact on winning Conference Championships. As I keep saying, the elite teams are less likely to play in the Super Bowl, but their chance of winning is mitigated by worse competition in the Super Bowl.

More importantly, Doug used a variant of PFR's Simple Rating System to establish team quality for his simulation (the stuff I used wasn't available in 2006). And rather than use an historical average, he produced the SRS each season with random numbers from a normal distribution (using a mean of 0 and a standard deviation of 6). This produces what appears to be the right range of SRS values (95% of teams between -12 and +12), but the bell curve is simply too flat.

Converting SRS to an equivalent team efficiency is simple, so I did that for historical SRS values from 2009-2013 and for a 10-year data set of SRS values that I randomly created from a normal distribution (exactly as Doug did). One more chart:

	SRS converted to GWP, 5-year mean	Random SRS converted to GWP, 10-year mean	Actual GWP 5-year mean
Team #1	74.1%	74.8%	76.2%
Team #2	71.8%	70.3%	73.6%
Team #3	69.0%	68.3%	70.8%
Average	50.0%	49.9%	50.0%
Standard Deviation	12.8%	12.1%	13.9%

His standard deviations are way too low. You can always define the parameters of a bell curve to fit the data of NFL teams, but in reality the quality distribution is not well described by a bell curve. It is much more linear. So his simulation fails to account for damage done to several elite teams who rank just below the league's #1 team.

Closing Thoughts

Can fans do math? I picture Goodell and the owners cackling maniacally at this analyis, confident that the unwashed masses have no chance of perceiving the degradation to the regular season and will remain entranced by broadcasts.

Really? Go get the dumbest football fan you can find and watch a few games with him. Does he know the difference between a 7-point lead, a 13-point lead, and a 17-point lead? Does he have a vague notion that an 11-5 team is better than a 9-7 team? Can he count that there are 12 teams remaining when the playoffs start, and just two wins needed for a Super Bowl trophy when the Conference Championship matchups are set?

Even casual fans are tuned in to the simple arithmetic required to understand football. They might not think of it as math owing to some mental block, but they understand the concepts of points, wins, and playoff rounds very well. If the playoffs are expanded, their perception of the importance of winning will quickly change for teams in the #2-#6 range, when a bye cannot be earned and a playoff spot cannot be lost.

Sure, most will still turn on the T.V. for their favorite team, even if they are vacuuming or playing video games or having sex while it's on. But the NFL relies heavily on television viewers who sit through as many as 5 games a week. How much will they pay attention when high stakes becomes no stakes?

Worse still is what happens when teams realize this. In 2005, the New England Patriots needed only a win in their final game to tie the Bengals with an 11-5 overall record and claim the tie-breaker with an 8-4 conference record. At stake was the #3 seed versus the #4 seed.

Belichick's response? The Patriots went out and tanked the closer against the Dolphins, pulling Tom Brady out of the lineup after a first-quarter refresher. It was a very astute move, considering that the "reward" for getting the #3 seed was an opening round game against the then-surging (and physically punishing) Pittsburgh Steelers.

Lest you think Belichick is the only one capable of this, the Bengals did exactly the same thing, pulling Carson Palmer after one quarter so they could lose to the Chiefs. Unfortunately for the Begnals, they had too many wins already, and backed into the 3-seed unwillingly. The #6-seed Steelers beat them up in the Wild Card round 31-17, while New England cruised past the #5-seed Jaguars 28-3.

That's what happens when there are no immediate stakes. Even without the very real possibility of deliberately losing to get a better first-round matchup, the frequency of injuries in the NFL gives teams a HUGE motivation to rest starters when they have little or nothing to play for. Right now, there is no immediate advantage for the #3 seed compared to the #4 seed, nor for #5 compared to #6. Expand the playoffs to 7 teams per conference, and you will have the same limitation comparing #2 vs #3, #3 vs #4, #2 vs #4, #5 vs #6, #6 vs #7, and #5 vs #7.

Seeds 3-5 even have a chance at a home game in the Divisional Round. If the playoffs are seeded perfectly (using the same GWP as the simulation), a typical #2 seed has a 79% chance to beat #7, and a #3 seed has a 68% chance of beating #6. The total probability of both teams winning is just 54%. Which means earning the #2 seed (instead of the #3 seed) will make a difference barely half the time. The difference between #3 and #4 is even smaller, and the differences from #5 to #7 are virtually nil.

Here's another very common scenario: Imagine that the Pittsburgh Steelers are sitting at 9-6 in the current format, and need a win in their final game to clinch a #6 seed (possibly #5 depending on other games). Pittsburgh is playing the 8-7 Ravens who have been mathematically eliminated. Will Pittsburgh play to win? Of course. Will the Ravens play to win? Hell, yes. Coaches like their jobs, player like their money, and everybody loves to play spoiler, especially against a division rival. It's must-see T.V.

Now take that same scenario with playoffs expanded to seven teams per conference. Pittsburgh at 9-6 is guaranteed a Wild Card spot no matter what, with only a variance in seeding (#5, #6, or #7) at stake. On the other hand, Baltimore at 8-7 is now in the hunt for the last seed. Will the Ravens play to win? Of course. Will Pittsburgh play to win? Hell, y--

No. Read that bit about the 2005 Patriots and Bengals again. Pittsburgh is almost certainly facing a 3-game road stretch to make the Super Bowl, and they would be fools not to pull their starters after the first quarter and take a mini-bye week with reduced injury and fatigue. Must-see T.V. just became a fiasco. Expansion will produce three times as many opportunities for playoff-bound teams to rest starters in the closing weeks of the season.

This will happen. Take it to the bank.

Lastly, Kenneth Arthur provides a juicy alternative for fans who want more playoff football:

What if instead of that, we don't just expand the number of playoff teams, but -- and Goodell will love this -- the number of playoff games.

Right now there are four rounds in the postseason with 12 teams and 11 games. In my 32-team system, there will be 263 games!

Take the 32 best NFL teams right now, force them to play a 16-game playoff system in which the 12 best teams move onto the final four rounds, culminating in a "Super Bowl" championship game to determine the winner. I know it sounds crazy but, hell, maybe it's just crazy enough to work.