Revisionism, Absolutism and Draft Grades

by John Morgan on Apr 28, 2008 9:59 AM PDT in Commentary Comment 27 comments

Update: I wanted to clarify something. I’m don't think looking back at past drafts, seeing what a GM values or overvalues, is at all a bad thing. I don’t think evaluating the overall success of a draft after a period of time is a bad thing either.

What I was speaking about is “Re-Drafts” and results based analysis. Beyond the inherent flaw, that being the inability of the person writing in the present to divorce themselves from all they know now and the team couldn’t know then, it’s also illogical because it assumes that had, say, another team drafted Frank Gore, that Gore’s career would have been identical except with a different team. Once you change any part of the past, all that follows is unknowable. Further, how players fit within a system, the talent of the other players they join, and how each influences the other, is inseparably linked to how we evaluate a player. If Joseph Addai hadn’t been drafted by the Colts, but instead by Kansas City or Baltimore, would we think Addai the same quality back?

So, with regards to what Mr. Rose and others have said, I’m not saying there isn’t value in evaluating a GM by his past drafts. I’m not saying their isn’t value in looking at drafts from the past, learning what worked, what didn’t. I’m only saying that for the commentator, it’s only fair to judge a draft when it happens. With the same ammunition the GM has. Otherwise, it’s not unlike telling me last week’s winning lotto numbers.

. . .

This nugget is making the rounds, so I apologize for picking on Mike Florio, who runs the indispensable Pro Football Talk , but I’ve stands as much as I can stands, and I can stands no more.

Every year at this time (actually, we can’t recall ever doing it before), we hand out our grades for the NFL draft.

Coincidentally, every team gets the same grade: Incomplete.

There’s no way of knowing for sure who will or won’t succeed at the next level until they enter the arena at the next level and show what they can do. If anyone knew for sure, the future Hall of Famers would always be the first few guys taken. Next would come the players who’ll make it to four or five Pro Bowls.

I completely disagree. Florio’s killing the exception and arguing absurdity:

So, because not every early pick turns into a HOFer, you cannot in any way evaluate a draft? That's insubstantial and misleading. It presents an either/or argument that implies either we know X for certain, or X is completely unknowable.

Lets take this to its logical end. If we really can’t grade a draft, than we can’t grade the individual picks within a draft. Then I certainly couldn’t argue something like this.

JaMarcus Russell did not deserve to be drafted first overall in 2007. He struggled to win the starting job at LSU, losing that position to future 7^th round pick Matt Flynn Russell's sophomore season. Russell was surrounded by pro caliber receiving talent. He also ran one of the more fundamental offenses in college football. Much of the hype surrounding Russell stemmed from his arm strength, a notorious red herring, and the way his team, LSU, dominated his chief rival’s team, Brady Quinn and Notre Dame, respectively. Russell was a known loafer, panicked under pressure and had trouble both learning an offense and reading a defense. It was a very risky pick and deserves a negative evaluation.

It's not any more unreasonable to evaluate any pick on available knowledge. How did a player perform in college? Was he a starter and for how long? What was the level of surrounding talent? What was the level of opposing talent? Has this person shown drive in all his pursuits? Consistent achievement? How was he evaluated by scouts? Known Red Flags: injury history, criminal history, conditioning problems, conflicts with coaches, etc. It doesn't mean you'll be right, but throwing your hands up and saying the draft is a crapshoot is nonsense. That most draft grades are given by people with poor analytical skills, bogus methods, or unwillingness to research and fact check doesn't mean that draft grades are in of themselves stupid or irrelevant.

Sorry, this is bordering on a screed, but absolutist thinking drives me nuts. For certain, a draft involves a great deal of uncertainty, but grades can be given and those grades can be founded on facts and logic.

The popular counterargument is that a draft can only be evaluated after 3 years or 5 years or whathaveyou. That, of course, is fundamentally flawed. You CANNOT judge a past draft, because you are privy to knowledge the people drafting could not have possibly known. It’s basic revisionism. In the 2006 draft, Marcus McNeill fell to the third round because he suffers from spinal stenosis. SS is a potentially debilitating condition, and teams were rightfully wary of drafting an offensive lineman with a potentially debilitating condition. His first season, McNeill went to the Pro Bowl.

Proponents of the revisionist argument took this chance to trumpet how inaccurate drafting could be. Except, those who questioned McNeill’s longterm viability were right then, and are beginning to look right now. McNeill allowed nearly twice as many sacks in 2007 (9.5) as he did in 2006 (5). The Chargers run blocking declined, and McNeill’s particular territory, left tackle, fell from 4th in adjusted line yards to 28^th . Anecdotal evidence buttresses these facts; McNeill looked stiffer and slower – as if he were suffering from a debilitating back condition.

A logical argument could be made, at the time of the draft, that McNeill was a far riskier pick than others in his class. No one questioned his ability. McNeill performed ably his first season but already looks to be validating concerns about his future. An intelligent grader should evaluate all these facts, including the inherent risk/reward of taking a potential franchise tackle in the 3^rd round, and provide an appropriate grade. An illogical critic can take that man to task because McNeill has a solid first season. Draft grades aren’t perfect, they’re too simple and obviously will hit and miss, but they are, done right, a reasoned, realistic assessment of a team’s draft when the draft happened. Revisionist grades are utterly worthless.

0 recs | Comment 27 comments

Story-email Email Printer Print

Comments

Wow...

Well done, John. That was an impressive rant. I agree completely.

by SeaTownBlueDevil on Apr 28, 2008 10:20 AM PDT reply actions 0 recs

I agree.

Off topic:

Dave Sims!!1
http://video.google.com/videoplay?docid=-699630719707539517&q=darius+reynaud&ei=lMwTSJWTBIqY-wHI3_GdBQ&hl=en
Schmitt happens.

by redwolf75 on Apr 28, 2008 10:39 AM PDT reply actions 0 recs

I read that post this morning as well and found it to be a complete waste of time. As for the contention that revisionist grades are totally worthless, I disagree. I think that revisionist grades are grades on organizations and scouting departments, and I think it is fair to evaluate how they preformed after the results are in. I think the McNeil example is the reason that waiting three years is the standard. Granted things like freak injuries can make a class look less than it would have been, but comparing what you thought before players perform and comparing it to what actually happened is a useful tool to help an organization refine and improve its strategies.

by SA77 on Apr 28, 2008 11:45 AM PDT reply actions 0 recs

Whats the difference between

Florio saying there is no way to know for sure, and you saying that the draft involves a great deal of uncertainty.

by Nate Dogg on Apr 28, 2008 12:27 PM PDT reply actions 0 recs

Simple, probabilty versus absolute.

I know the draft involves uncertainty, but I won’t say it is impossible to evaluate or that draft grades are worthless.

by John Morgan on Apr 28, 2008 1:00 PM PDT up reply actions 0 recs

But you absolutely can't tell me exactly what any of these player will do

I agree with what you’re saying but disagree with the point you’re trying to make. You can give me a probability that the Chiefs will come out with a more successful draft than the Dolphins, sure, but what does that even mean? It’s just a guess, an educated guess sure, but just a guess.

I’d rather wait 3 to 5 years and look back and try to judge based on actual performance rather than probable performance. Of course you have to take special exceptions like Marcus McNeil into consideration, but it’s not revisionist to go back and say that some scouts dropped the ball on players like Tom Brady or Gerrard Warren.

I’m curious what you think the value of day after draft grades are.

by Nate Dogg on Apr 28, 2008 1:27 PM PDT up reply actions 0 recs

I agree, good post JM

Too much of what is considered “success” is results driven. So, say a guy goes 0 for 4 in a baseball game, failure right? Well, if two were line drives that happened to be caught, one was a groundball up the middle that a great defensive play stopped, and the other one was a popup that took 10 pitches, then all the ABs were successful. Same applies the other way, if a guy goes 4 for 4, but only sees 6 total pitches, and 2 of the hits were lucky fly balls that bad outfielders misplayed (but not enough for an error) and the other 2 hits were dinky hits like Frank Thomas’ last night, then the results say he was successful, the path to the results say he was lucky.
As for your Marcus McNeil example, I think that’s a great example. Yes, he has the tools, talent, and work ethic, but if everything goes right with no chance of injury, he’s only an 80% chance of sucess, and skew towards how great he can be matters as well, but an injury such as the one he has, might give him a 50/50 chance of not playing, and that has to be considered. Sure he ended up having a great season, but he’s still a risk, and teams in general recognized that risk and rated him on their draft boards accordingly.

http://seattlesportsmaniac.blogspot.com

by LantermanC on Apr 28, 2008 1:02 PM PDT reply actions 0 recs

Exactly

Grades have to be given based on what each organization knew at the time. That said, I do think there’s value in giving grades three or five years after a draft, too, as long as you factor out what they could not have known at the time. For example, for instance, you hated NE’s pick of Mayo because you didn’t think he fit their system; obviously, Belichick/Pioli think otherwise, and only time will tell who was right. But I agree completely that purely results-driven “analysis” isn’t worth much.

by The Ancient Mariner on Apr 28, 2008 1:28 PM PDT reply actions 0 recs

How is analysis based on

actual results worth less than analysis based on conjecture and expectation?

by Nate Dogg on Apr 28, 2008 1:38 PM PDT up reply actions 0 recs

It's not analysis at all.

It’s saying “wow this worked out, they must have known what they were doing”. It has no analytical or predicative value.

by John Morgan on Apr 28, 2008 1:39 PM PDT up reply actions 0 recs

Well...

If a team consistently obtains good results from decisions that are generally thought to be poor, then perhaps “the horde” has to rethink it’s methodology for making these assessments in the first place.

by cyberwulf on Apr 28, 2008 1:43 PM PDT up reply actions 0 recs

It's = its

I hate when other people screw that up, so now I’m angry at myself for doing it.

by cyberwulf on Apr 28, 2008 1:44 PM PDT up reply actions 0 recs

Or perhaps the team simply got lucky

Good results do not necessarily imply good decision making

by ningwers on Apr 28, 2008 1:48 PM PDT up reply actions 0 recs

But you are correct that it does warrant a closer look

by ningwers on Apr 28, 2008 1:48 PM PDT up reply actions 0 recs

It has predictive power

in that you can learn from mistakes and success’s to get a better idea of what makes an NFL player. Giving the Jaguars an A because they solved their pass rushing needs with Quentin Groves and Derrick Harvey doesn’t have any predictive power, it’s just an expectation. Going back and seeing why Jason Taylor slid to the third and what it was that made him better than people predicted is valuable.

Day after draft grades say far more about the grader than it does about the actual picks. Someone like yourself, who tends to be a high floor / low ceiling type of evaluator, is going to grade a draft far differently from someone who is a high risk/reward type evaluator. You can argue all you want about the value of either mind set, and I’d most likely agree with you, but you can’t say anything for sure about the player you’re discussing until he gets out on the field.

by Nate Dogg on Apr 28, 2008 2:08 PM PDT up reply actions 0 recs

I think the thing you're missing is that the type of revisionist evalution that John is talking about

generally involves looking at the results without looking at the decisions that went into the draft selections or into the player evaluations. They simply say Player A is good now, therefore the team that drafted him had a good draft. Player A could’ve been selected simply because he had a cool haircut, and his team would receive a better revisionist draft rating than a team who made well reasoned selections but simply got unlucky.

by ningwers on Apr 28, 2008 2:15 PM PDT up reply actions 0 recs

The same logic

applies to trades. You can evaluate a trade at the time it’s made, and argue about whether or not one team got the short end of the stick. Of course, all such assessments involve subjective judgments based on incomplete information.

In contrast, looking at a trade in retrospect involves no debate; based on the performance of the players involved, it is obvious which team has benefited and which has suffered.

Revisionism is, in some sense, a way that prognosticators can see whether their methodology for making initial assessments is flawed.

by cyberwulf on Apr 28, 2008 1:41 PM PDT reply actions 0 recs

Hey John and Field Gulls nation, Ben Riley here from Football Outsiders. First off, thanks to everyone at FG for the fantastic work leading up to the draft. John, you were singlehandedly responsible for making me sweat bullets over the mere possibility of “With the 25th pick, the Seahawks take Kentwann Balmer, overhyped one-year wonder…” Instead, he went to the Niners. Ha.

As to your post-draft rant about draft grading, I am in 100% agreement with you that there’s merit to evaluating drafts immediately after they are completed. Like some of the commenters, however, I’m surprised you make the strong claim that “revisionist” grades are “utterly worthless.” Obviously, luck and circumstance play a large part in any draft, but over time certain patterns do emerge in a particular general manager’s skills and weaknesses at talent evaluation. (Who might I be thinking of here?) A revisionist draft grade - perhaps historical is a better adjective - is the only way I know of to detect those patterns. (And if you think I may be previewing a possible theme of the Seahawks chapter in the upcoming PFP:2008, hey, I owe you a Jones “Turf flavor!” soda.)

by bjr on Apr 28, 2008 7:24 PM PDT reply actions 0 recs

Some much deserved praise for John

by Nate Dogg on Apr 28, 2008 9:21 PM PDT up reply actions 0 recs

cool

by clamslayer on Apr 28, 2008 10:34 PM PDT up reply actions 0 recs

What the fuck are you guys talking about?

Did I just take some acid and wander into the desert? What the fuck is going on here?

Are people really debating whether it’s okay to debate? Evaluate em, criticize em, taunt em, tickle em, fuck em in the ass.. who cares? You put it out there, we’ll soak it up.. One year, two years, three years – NFL fans are like crack addicts with a substantially higher income and less common sense.. give us more, quality is out the window, just give us more..

by BS! on Apr 28, 2008 9:15 PM PDT reply actions 0 recs

Reductio ad absurdum

is a type of logical argument where one assumes a claim for the sake of argument and derives an absurd or ridiculous outcome, and then concludes that the original claim must have been wrong as it led to an absurd result. Wikipedia at (http://en.wikipedia.org/wiki/Reductio_ad_absurdum)

I don’t really see how what these folks are doing, your “revisionist” label, as being an example of appeal to absurdity.

I don’t think they’re claiming as much as you think they’re claiming. They’re just choosing to magnify the unknown risk of how players will perform, because in the end, all players are judged by their on-field results rather than their potential on draft weekend. Many things are completely of their control from draft weekend until the end of their career.. freak accidents, etc.

It’s like blackjack. A player who does not hit a hard 16 when the dealer has a 10 card or ace showing is at a statistical disadvantage in the long run compared to a player who hits it. If you try and grade a draft right now before the dealer shows their hole card (the player’s career, in this metaphor), then your judgment will be unduly biased towards the statistical potential rather than the real result of that hand. You’re saying that the statistical potential is all that matters, i.e., if 6th rounders go on to have HOF careers, teams should not receive extra recognition retroactively for their judgment because they didn’t really know they would have a HOF career or they would have drafted them earlier.

Even though statistics point to hitting a hard 16 against a dealer’s 10 card, in a certain hand it may have actually been better to stay if the dealer were to go on to bust. Your argument is that the player who superstitiously stays on a 16 should not get credited for making the right decision when the rest of the hand is dealt because what the player really faced as a decision based on those statistical odds and they made the wrong one.

I disagree. I’ve seen too many people get lucky. The Pats sure wouldn’t be who they are today if it weren’t for Mr. Brady and there are innumerable better examples than that.. They deserve “revisionist” recognition for picking the guy who turned out to be a HOFer.

by clamslayer on Apr 28, 2008 10:50 PM PDT reply actions 0 recs

here's my opinion

I think a large problem with immediate draft grades is that you are evaluating the picks of franchises that have access to way more information then you do. A lot of what it takes for a player develop into a good nfl player or a scrub is at this point unquantifiable. And a lot of what makes a front office make a decision about whether to draft a player we simply do not have access to. So draft grades are made from a position of relative ignorance. I’m not saying they aren’t valid, but I don’t think they are the end all and be all.

There is valid in this so called “revisionist” grading, if a team is consistently getting better results out of their players then the rest of the league, it has to be attributed to more then just luck. There will always be a single player that comes from nowhere and tears up the league, and that IS most likely luck, but there are certain teams that are able to squeeze value out of their draft picks more consistently then the rest of the league. I imagine, if you look back at their immediate draft grades, they aren’t going to be above the rest of the leagues. To me that shows that these teams are either able to develop their talent better, or are better at recognizing it in the draft, and I think revisiting drafts of teams is a good way to investigate that.

by gumbostu on Apr 29, 2008 12:10 AM PDT reply actions 0 recs

the problem is more the "expert" evaluators

than the process, or timeline.
Everyone bangs the manning/leaf pot, but really, the true mark is reggie bush/mario williams.
The texans were killed for taking mario. Bush was a sure hit, constant threat, saviour of the Saints. Now he is viewed as unable to be an every down back, unable to block, doesn’t have long gains, etc. etc. And opinions may flipflop again this season.

The point is that grades are often flawed because:
(1) they are kneejerk reactions
(2) the people giving them are not learned enough to give
(3) not enough research or preparation goes into the reviews
(4) when you have several hundred guys, 32 teams, trades, veterans, rule changes, salary caps and other variations, the sheer process of evaluating a player, team or the overall draft is a tremendous undertaking with myriad crosswalks, far too large for any one person to take up.

That said, there certainly are picks that look bad or good, but there is no accountability from “experts”. Anyone can say anthing about a player being good or bad now or down the road and it will not matter.

by vherub on Apr 29, 2008 9:08 AM PDT reply actions 0 recs