AI Now Solves 97% of Coding Benchmarks and Fails 77% of Real Software Work. Both Numbers Are True.

Last week a developer sent me a screenshot of a benchmark leaderboard with a one-line message: "Should I even bother finishing my TypeScript course?"

The leaderboard showed Claude Opus 5 resolving 97% of tasks on SWE-bench Verified, the industry standard test for AI coding ability. GPT and other frontier models clustered right behind it, most of them above 95%. To someone learning to code in 2026, that number reads like a death sentence. If the machine solves 97% of software engineering problems, what exactly is left for a human to do?

I understand the panic. I get some version of that message every week now. But I want to walk through what that 97% actually measures, because right next to it in the research is another number almost nobody is talking about, and that second number tells the real story.

On a newer benchmark called SWE-EVO, the same class of models that scores in the nineties on SWE-bench drops to roughly 23%. GPT-5.2, which resolves 72.8% on SWE-bench Verified, falls to 22.9% on SWE-EVO. Not a small decline. A collapse. Same model, same underlying intelligence, and the resolution rate falls by two thirds.

Both numbers are real. Both are measured on the same models. The gap between them is the most important thing happening in software engineering right now, and it is where your entire career is about to live.

What the 97% Actually Measures

SWE-bench is built from real GitHub issues in real open-source repositories. That sounds like real work, and in a narrow sense it is. But look at the shape of the task. The model is handed a single, well-defined issue with a clear description and a reference to the specific buggy code. Its job is to produce one patch that makes the existing test suite pass. Problem in, patch out, graded against tests that already exist.

That is a genuinely hard thing to do well, and the fact that AI does it at 97% is a real achievement. I am not here to diminish it. If your job consists entirely of picking up perfectly specified tickets and producing isolated patches against existing tests, then yes, you should be concerned, because that specific slice of the work is now largely automatable.

But here is the thing almost nobody who panics at the 97% number understands. That slice is a small fraction of what software engineering actually is. The research is explicit about this. In practice, up to 80% of engineering effort goes into maintaining and evolving existing systems, which requires coordinated changes across modules, versions, and specifications. SWE-bench does not test that. It tests the 20%, the discrete, isolated, well-specified 20%, and it tests it very well.

The 97% is not a measure of how much of your job AI can do. It is a measure of how well AI does the cleanest, most isolated, most artificially bounded fraction of your job. Those are completely different claims, and the gap between them is enormous.

What Happens When You Test the Other 80%

This is why SWE-EVO matters, and why I think it is the most important coding benchmark released in the last year even though almost no one outside research circles has heard of it.

SWE-EVO does not hand the model a clean ticket. It hands it a release transition. The task is constructed from what actually happens when a real software project moves from one version to the next: interpret the release notes, implement comprehensive changes that resolve multiple pull requests at once, develop new features, and fix multiple bugs, all while keeping the existing system working. The average task spans changes across 21 files and is validated against test suites averaging 874 tests per instance.

That is real software engineering. Not a ticket. A system in motion.

And on that benchmark, the models that look superhuman on SWE-bench fall apart. The best model reaches only 25%. GPT-5.2 drops to 22.9%. The intelligence did not change between the two tests. What changed is that SWE-EVO requires something the isolated benchmark never asked for: sustained, multi-file reasoning about a system that has history, dependencies, and context that is not written down in any single ticket.

The AI can fix the bug you hand it. It cannot yet hold the whole evolving system in its head, understand why the previous version was built the way it was, anticipate what a change in one module breaks three modules away, and coordinate a release that keeps 874 tests green. That is not a small gap. That is the difference between a very fast intern and a senior engineer, and right now the machine is firmly on the intern side of that line.

Why This Gap Is Specifically a JavaScript Problem

I want to make this concrete for the people who read my board, because the benchmark story lands differently depending on your stack, and for JavaScript and TypeScript developers it lands in a specific and useful way.

Notice something in the benchmark data. Most of these tests, SWE-bench and SWE-EVO both, are built on Python repositories. When researchers do measure performance across languages, JavaScript and TypeScript consistently come out messier and harder for the models, with resolution rates that swing wildly from near zero to the low thirties depending on the specific codebase. There is a reason for that, and it is not that the models are worse at JavaScript syntax.

It is that JavaScript codebases in the real world are evolution problems almost by nature. A mature front-end application has years of accumulated architectural decisions, three different state management approaches from three different eras, a component library that was half migrated to a new version and then abandoned, and framework choices that made sense in 2021 and are now load-bearing technical debt. This is the exact environment where SWE-EVO-style reasoning is required and where AI currently fails. The messiness that makes JavaScript codebases painful to work in is the same messiness that AI cannot yet navigate.

That messiness is your moat. I have written before about why system design is the one skill AI cannot automate, and this benchmark gap is the empirical proof of it. The developer who can hold a sprawling React application in their head, understand why the authentication flow was built the way it was, and coordinate a migration across twenty files without breaking the eight hundred tests that guard it, is doing exactly the thing the best models score 23% on. That skill is not depreciating. It is appreciating, because the tooling that surrounds it is getting cheaper while the skill itself stays scarce.

The Trap of Reading the Wrong Number

Here is where I want to push back on the dominant narrative, because I think a lot of very smart people are drawing exactly the wrong conclusion from these numbers.

The panic reading of the 97% says: AI is coming for all of it, so specialize in AI, learn to prompt, become an operator of the machine. The problem with that reading is that it optimizes for the part of the job the machine is already best at. If you spend the next year becoming excellent at handing well-specified tickets to an AI and reviewing isolated patches, you have trained yourself for the exact slice of work that is most automated and least valuable. You have run toward the 97%, not away from it.

The number you should be optimizing for is the 23%. The work AI fails at is the work that pays, and it will stay the work that pays precisely because it is the work that resists automation. Multi-file reasoning. System evolution. Understanding why something was built before deciding how to change it. Holding context that lives in no single ticket. These are not glamorous skills. They do not have a certification or a bootcamp module. They are built slowly, by working in real codebases and paying attention to why things are the way they are. And they are the durable center of an engineering career in the age of AI.

I said something similar when I wrote about the ghost technical debt that AI agents leave behind, and the benchmark data confirms it from another angle. When companies deploy AI to generate code at the 97% level without humans who can operate at the level SWE-EVO measures, they are manufacturing exactly the kind of unmaintainable, context-free, multi-file mess that the machine itself scores 23% on cleaning up. The generation is cheap. The evolution is expensive. And the people who can do the expensive part are about to be worth a great deal.

What I Actually Think You Should Do

Let me be direct, because the developer who messaged me deserves a real answer, not a reassuring platitude.

Finish the TypeScript course. But understand that the course is the floor, not the ceiling. Learning the language and the framework gets you to the level the machine already matches. What separates you from the machine is everything the course does not teach: how to read a large unfamiliar codebase and understand its history, how to reason about what a change ripples into, how to evolve a system rather than just patch it. Those are the skills that live in the gap between 97% and 23%, and they are the only skills worth building a career on right now.

Stop measuring yourself against the benchmark the machine wins. A human competing with Claude Opus 5 on isolated, well-specified GitHub issues is competing on the machine's home turf, and will lose. That is fine. That was never the valuable work. Compete instead on the thing the benchmarks prove the machine cannot do, which is holding an entire evolving system in your head and making it better without breaking it.

The two numbers are both true. AI solves 97% of the clean, isolated, artificial version of software engineering, and fails 77% of the messy, evolving, real version. Your career is the second number. Build for it deliberately, and the first number stops being a threat and starts being the most powerful tool you have ever had.

FAQ

Does AI scoring 97% on SWE-bench mean software engineering jobs are disappearing? No. SWE-bench measures isolated, well-specified tasks: one clear GitHub issue, one patch, graded against existing tests. That describes roughly 20% of real engineering work. On SWE-EVO, a benchmark that tests system evolution across multiple files and versions, the same models drop to around 23%. The high benchmark score reflects how well AI does the cleanest slice of the job, not how much of the job it can actually do.

What is the difference between SWE-bench and SWE-EVO? SWE-bench hands a model a single well-defined issue and asks for one patch that passes existing tests. SWE-EVO hands it a full release transition: interpret release notes, resolve multiple pull requests, add features, and fix bugs across an average of 21 files validated against roughly 874 tests, all while keeping the system working. The first tests isolated problem-solving. The second tests sustained, multi-file reasoning about an evolving system, which is closer to real engineering.

What skills should JavaScript developers build given these benchmark results? Focus on the work AI fails at: reading large unfamiliar codebases and understanding their history, reasoning about how a change ripples across modules, and evolving a system rather than patching isolated issues. These multi-file, system-level skills are what the best models score around 23% on, which makes them the durable center of a JavaScript career as isolated coding tasks get automated.