Based on some commentary in this thread, I became interested in exploring whether small-school players have tended to be over- or under-rated on draft day. Basically, the question is:

Do players coming from non-BCS schools perform better/worse in the NFL than comparable big-school (BCS) players?

LantermanC was kind enough to provide me with his awesome Excel spreadsheet containing every player drafted from 1984 through 2007, along with position played, games started, Pro Bowls, and a plethora of other information. We'll be looking at two outcome variables, compiled by him:

Success = 1 pro bowl or 3 starting seasons, or 64 games played.

Wild Success = 2 pro bowls, 7 starting seasons, or 125 games played.

I restricted my analyses to the years 1993 (when the draft went to the current 7-round format) to 2005 (though data is available through 2007, recently drafted players haven't generally accumulated enough stats to be classified as successful).  Players were categorized as being "small-school" or "large-school" according to whether they played for a BCS or non-BCS college. Undrafted players were excluded.

The knee-jerk reaction to answering the initial question is to compare the Success and Wild Success percentages of BCS and non-BCS players. But non-BCS players tend to be drafted in later rounds:

Average pick, BCS players:116

Average pick, non-BCS players:147

It's not a fair comparison. So, what we'd like to do is to somehow adjust for draft position. A more refined version of our initial question is then:

Given a BCS and non-BCS player selected at the same spot in the draft, which has performed better in the NFL?

Fortunately, we can answer this question using a basic logistic regression. Without going into too much detail, we model the probability of Success (or Wild Success) as a function of BCS/non-BCS and draft position. This allows us to compare the success probability of BCS and non-BCS players given that they were selected at the same place in the draft.

Running the analyses, we get that:

1. Adjusting for draft position, players from non-BCS schools are roughly 4% less likely (95% confidence interval: 20% less likely, 15% more likely) to have Success than BCS college players

2. Adjusting for draft position players from non-BCS schools are roughly 1% more likely (95% confidence interval: 20% less likely, 29% more likely) to have Wild Success than BCS college players

I re-ran the analyses, adjusting for position played; the results are basically identical.

So, bottom line:

There is little evidence to suggest that non-BCS players exceed or fall short of their expected performance based on where they were drafted.

In other words, NFL GM's do not appear to have systematically downgraded (or upgraded) players based on whether they attended a BCS or non-BCS college.

Final notes:

- If small-school players are systematically denied playing time (due to their lack of pedigree) when they have reached the NFL level, then this analysis could be misleading. In fact, if this is the case, you can't even answer the initial question.

- This analysis refers to overall trends in the NFL; it doesn't claim to be able to predict a specific GM's behavior in the future.  However, it does seem to indicate that there is no substantial "market inefficiency" that can be easily exploited.

Interested to hear what people think. Thanks again to LantermanC for the fantastic dataset.