You are not falling behind. You are watching the wrong scoreboard.

The scoreboard that changes every six weeks

Here is what the last ten days sounded like if you are an engineer with a job and a family and a finite amount of evening.

Theo Browne released a video saying that if you cannot feel the difference between this generation’s frontier models and the last one, you suck at prompting, and he meant it kindly but he meant it. ThePrimeagen spent an episode on how a company that invented the transformer and the tensor processing unit is, on the coding leaderboards he showed, being beaten by a lab with a few hundred employees, and then left one of its models on a trivial bug for forty minutes and watched it spend 330 million tokens reading the same file. AI Explained walked through six axes on which the labs say capability will keep improving, none of them near saturation, and quoted a researcher saying there is a large gap between how fast progress looks from inside and from outside. A model called Astra that could do things nothing before it could. Another called Fable that was the safe choice three weeks ago. A Chinese open-weight model that is months old and beating both on some table.

If you tried to keep up with that, you did not sleep and you learned nothing durable, because almost every specific fact in the paragraph above will be wrong by November. That is not a prediction about any particular model. It is a description of the scoreboard. It reshuffles every six to eight weeks and it has done so for two years and every lab in the AI Explained video says it will keep doing so.

So the anxiety is real and the scoreboard is real and the two have almost nothing to do with each other. The question worth an evening is which things you can learn now that will still be true when the scoreboard has reshuffled four more times.

The half-life of what you know

Sort what an engineer learns about these tools by how long it stays true.

At the bottom, decaying in weeks: which model is best at which task, which one reads the same file twenty-six times, which one is over-eager and which one is lazy, the prompt phrasing that makes a particular model stop apologising. This is what most of the content is about, because it changes constantly and change is what content is made of. Learning it is not worthless. Learning it is maintenance, like knowing this quarter’s prices.

In the middle, decaying in months: the harness. Which tool, which agent runtime, which instruction file format, which permission model. The study we wrote about earlier this week found that the harness is a first-class variable, more important than the leaderboards admit, and that is true. It is also true that the harnesses are being rewritten as fast as the models, and that the changelog entry that got seven hundred points this week was about one tool starting to read another tool’s config file.

At the top, not decaying at all as far as anyone can tell: five things. We will name them and then argue for each one, because the argument is where the reassurance actually lives.

The five things

Specification. The single skill that has appreciated most in two years is the ability to say, precisely and in writing, what a system should do, including what it should refuse to do and every exception. This used to be a skill people apologised for having, because the code was the specification and writing it down twice was waste. Now the specification is the input to the machine that writes the code, and the quality of the output is bounded by the quality of that input in a way that no model upgrade changes. Every team that got disappointing results from agents this year got them from underspecified tasks. The detail.dev post says the most valuable engineering work is going to be having good ideas, and an idea that cannot be specified is not yet an idea.

Verification. Knowing whether a thing works, and being able to prove it to someone else, is the skill the machines are worst at and the one they generate the most demand for. Every agent-written change needs someone who can say what test would catch the failure, whether the test that was written is that test, and whether the green result means what it appears to mean. Dan Luu’s essay this week is on the front page for a reason: there is no point at which turning your brain off works, and verification is the name for the brain being on. This is also, not coincidentally, the skill Theo was actually describing when he said people prompt badly. The people who get good results from strong models are the people who can tell when the result is bad.

Judgement. Which of the forty things the model could do is the one worth doing. Which of the three approaches it offered is the one that will still be fine in a year. When to stop. When the obvious solution is obvious because it is wrong. Every essay on this site comes back to this because every failure we have watched comes back to it: the model did something competent that nobody should have asked for. Judgement is slow to build, does not transfer from a video, and is the entire reason a senior engineer is paid more than a junior one with the same tools.

Domain intimacy. Knowing the business, the customers, the forty exceptions and why the freight claim in 2023 changed how one account is handled. We have argued at length that companies do not have processes, they have habits, and that the habits live in heads. The heads are the moat. A model with the rumour builds the obvious version of your product; the person who knows why the obvious version fails for the second-largest customer is the person the swarm cannot replace, and that person is more valuable this year than last, not less.

The seat. The ability to occupy the approval seat well: to look at a model’s output, know where to spend eight seconds, sign for it, and stand behind the signature. This is the one that combines the other four and it is the one the organisation will pay for when the volume of model output has outrun everyone’s ability to read it. It is also the one nobody teaches, because until eighteen months ago the person in the seat had also done the work.

What this means for the evening

Spend it differently.

Stop trying to keep up with the scoreboard and start reading it the way you read exchange rates: glance, note the direction, move on. Pick one strong model and one harness and learn them properly for a quarter rather than four of each badly. Theo’s floor-and-ceiling point is correct and it is also an argument for depth: you learn where a model’s floor is by using it on dull tasks for weeks, not by watching someone else’s demo of its ceiling.

Then spend the real time on the five. Write the specification for the next thing your team builds before anyone, human or model, writes a line, and notice how much you did not know. Write the test that would catch the failure before you look at the agent’s tests. When an agent offers three approaches, write down why you picked one, and read your reasons back in a month. Learn the part of the business your team pretends is simple. Volunteer for the review seat that everyone else is avoiding because the queue is long, and get fast at it, because that queue is the most important place in the company and almost nobody wants to sit there.

The honest caveat

It is possible to hide from the tools behind this list, and some people will. Specification and verification and judgement are worth nothing if you refuse to use the machines that make them valuable, and the engineer who has not sat in front of a frontier model for a hundred hours does not actually know where its floor is and cannot occupy the seat. The scoreboard is not the point, but the tools are, and the argument here is for depth with them rather than distance from them.

The researchers in the AI Explained video may be right that the gap between inside and outside is large and that things will move faster than the outside expects. If so, the scoreboard will reshuffle faster, not slower, and the half-life of everything in the bottom two rows gets shorter. The five things at the top do not get shorter. They get more expensive, because the volume of machine output that needs specifying, verifying, judging, situating and signing for goes up with every release, and the number of people who can do those things well does not.

You are not behind. You have been watching the part of the screen that changes. Look at the part that does not, and get good at it while everyone else is refreshing the leaderboard.