What a grade cannot tell you

Two weeks ago I made three claims about how programs below the money decide whether to develop a player or buy one. I said I would test them this season and publish what I found, including if I turned out to be wrong.

Before anyone else attacks the test, I want to attack it myself.

The obvious way to check whether a develop bet paid off is to look at how the player performed. The obvious way to look at how a player performed is his grade. Almost every program now has one, from a grading service or its own staff, and it has become the closest thing the sport has to a common currency for player value. Coaches cite it. Portal evaluators screen with it. The people pricing players lean on it more every year.

A grade is a good instrument. It measures one thing well. The trouble is that the thing it measures is not the thing my claims need.

What a grade actually measures

A grader watches a snap, decides what each player was supposed to do, and scores how well he did it. Win your block, positive. Get beaten, negative. Over a season those snaps add up to a number.

That number answers one question: given the job he was handed, did he do it?

It does not answer whether it was the right job.

Here is a snap every line coach has watched. Third and seven. Before the snap, the defense shows pressure to one side, and the protection slides to meet it. At the snap, the pressure drops out and comes from the other side. The right tackle is now left with two rushers and one pair of hands. Nobody in the building thinks he had a chance. He gets beaten, and the quarterback gets hit.

The tackle grades negative. The call that put him there does not grade at all.

Now reverse it. A staff sees a young guard struggling with one particular stunt and quietly stops calling the plays that expose him. His grade looks fine. What he cost the offense never appears on his grade. It appears as the plays that were not called.

Execution inside a bad call grades badly. Execution inside a protective call grades well. In both cases, the grade is recording a staff decision and assigning it to a player.

That is not noise. Noise washes out over a season. This does not, because the calls are not random. The same people make them, with the same tendencies, every week.

What that does to my own claims

Apply that to what I argued two weeks ago.

Claim one said the develop path is overvalued because it hides the cost of the season you pay while you wait. To price that season, I need to know how much worse the project played than the ready-now alternative would have. The first place anyone would look is the grade gap.

But a staff that knows it is playing a project does not call the same game. It protects him. It shrinks the call sheet around him. It moves help his way. Part of the cost of development never reaches his grade. It moves to the call sheet, where no grade can see it.

So the grade undercounts the cost when the staff protects the player and overcounts it when the staff exposes him. Which way it errs depends on decisions nobody is grading. Tested on grades alone, my first claim could come out right or wrong for reasons that have nothing to do with whether it is true.

Claim two said development success is a liability, because a player who becomes worth more than you pay him becomes worth more to somebody else. The market decides what "worth more" means, and the market reads grades. A player whose grade was lifted by good calls carries that grade into the portal. The staff that built a plan around him raised his price for every other buyer. Part of the exit risk I described is created by the coaching, not the player, and a grade cannot tell the two apart.

Claim three survives this, and I want to be clear about why. It rests on who stayed and who left, which is roster data, not grades. Whether it holds up is a separate question. It is just not exposed to this problem.

Two of my three claims cannot be tested the way most people would test them.

What the grade misses, measured

To see how big this problem is, I graded one offensive line through one Group of Five game from this season. I didn't have the call sheet, which is the position every outside grader and portal evaluator is in.

Of 63 snaps, the line lost 18. I then went back through those 18. Seven were clearly on the player: the assignment was clean, the structure was sound, and he got beat. The other 11, to my eye, were not. The loss traced to how the play was built, with an unblocked rusher, one blocker left with two defenders, or a protection sliding away from the pressure, or there wasn't enough on film to say it was his fault.

That's 11 of 18 losses, more than half, that would have gone into a lineman's grade as his failure when the film can't show that it was.

It's one game, one line and one grader, and I wouldn't lean on the exact number. The direction is the point. A buyer reading that grade is partly reading the offense the player played in, and he has no way of knowing how much.

The question grades were never built to answer

It would be easy to read this as an argument that grades are wrong. They are not. A grade does exactly what it was built to do, which is score a player against the job a grader believed he was given, in a system the grader did not design and cannot see.

The trouble starts when that number gets asked to settle a different question. Not how did he perform, but can this player do the specific job we are going to ask him to do?

Those are not the same question, and only the second one spends money. Every portal decision, every development decision, every dollar of a roster budget is a bet on the second one. Almost all of them are made on evidence built for the first.

Anyone who has been in football knows how this actually works. Players win games. The staff's job is to put them in position to win them. Both halves of that sentence are true, which is exactly why a number that only records the first half is a poor instrument for deciding who to pay. A high grade earned in a system that protected a player's weaknesses is not a promise that he will hold up in yours. A low grade earned on an island against an unblocked rusher is not proof he cannot.

The call is what separates the two, and every staff in the country already has it. It is on the call sheet, in the self-scout, in the weekly breakdown. It gets used for game planning and then filed. It almost never sits next to the grade, snap by snap, for the same plays — and lining the two up is harder than it sounds, because they were never built to meet.

Put them together and you stop grading performance and start measuring fit: how a player performed relative to the difficulty of what he was asked to do, in the jobs he was actually asked to do. That is a number you can compare across players, against a role, and eventually against what he costs.

What happens next

If a grade cannot separate the player from the job he was handed, then one of football's oldest debates, talent or scheme, cannot be settled with the evidence most programs have.

That is the next piece, and I will say now where I expect it to land. Not on either side. Football is won by a team: players executing, coaches putting them in position, and a lot happening in between. Splitting credit or blame between the two was never the useful question.

The useful question is how well the pieces fit. Which players can execute what this staff wants to call, where the plan and the personnel line up, and where the money is buying the most of both. That is not about who was wrong last Saturday. It is about getting more out of the same roster and the same budget next Saturday.

Every program is already trying to do that. Most are doing it without the evidence to know whether it is working.

Next in this series, October 8: horses or jockey.  All research →