We put Claude Sonnet 4.6’s AI code quality to the test
In this article
Summary
We ran a controlled experiment on Claude Sonnet 4.6, building the same system 20 times with one variable changed: whether a real-time, deterministic quality and security check sat inside the agent’s loop.
The results:
- High-risk security findings dropped by roughly 97%.
- Maintainability score rose by roughly 24%
- Every guided run scored higher on maintainability than every unguided run. The worst guided build beat the best unguided one.
- By catching problems before they trigger rounds of regeneration, the experiment points to more efficient token use, not more expensive.
Below, we explain what we measured in more detail, how it connects to patterns we have seen before, and where the Model Context Protocol (MCP) fits in to support the Agentic AI software lifecycle.
The quality of AI code has improved
AI code quality has improved a lot. It is genuinely impressive. Some of the newer models excel at generating new code, and the quality shift has been enormous. We have all watched the mood change in real time. People who were loudly skeptical a year ago are suddenly a lot quieter, and some are fully won over.
The question I’m trying to answer in this article is what happens once capable models are dropped into an agentic loop, asked to build something end to end, with nobody checking its work until it is done.
Does the same quality hold up at speed, unsupervised, across a full build?
This is the question we hear most from engineering leaders right now. Agents clearly write code faster. Does the code that comes out the other end hold up? Not “does it run,” but does it stay maintainable, secure, and safe to build on six months from now?
We could have kept guessing. Instead, we ran an experiment using Sigrid®’s Guardrails, an MCP built by Software Improvement Group (SIG) for AI coding assistants and agents that checks maintainability and security, right as the code is generated in the Integrated Development Environment (IDE).
How to stay in the loop, when the loop moves too fast
A coding agent that works fast and a coding agent that works well are not the same thing, and in 2026 it is easy to mistake one for the other.
It’s easy to think we just need to keep a human in the loop to stay in control. But what happens when the loop starts moving too fast for humans to keep up?
In a recent episode of the SIGNAL podcast, I spoke to Luc Brandts about this, and he brought up the difference between a calculator and AI to make a very important point.
“The difference between a calculator and AI is that AI will give you a different answer every time” – Luc Brandts, Agentic AI in software engineering SIGNAL podcast.
You can’t rely blindly on the decisions an AI makes. So, you need to put something predictable in there so that you can.
How? It’s possible to ground your agents so that they check each change against deterministic (predictable) signals, inside guardrails you set up. All via the Model Context Protocol (MCP).
To prove this works, we put our Sigrid Guardrails to the test and compared the output of one of the most popular models used for AI coding assistance: Anthropic’s Claude Sonnet 4.6.
How did we put Claude Sonnet 4.6 to the test?
We built the same project management system 20 times with Claude Sonnet 4.6, changing exactly one thing between runs. We had two arms, a control group and a treatment group.
- In the control group, the agent worked from the same upfront coding instructions any well-configured setup would get: a Code Principles block describing the standards the output should meet. Nothing else.
- In the treatment group, the agent received the identical upfront instructions, plus one addition: Sigrid Guardrails, wired into the build loop through a mandatory quality and security gate. It had to build, run the quality checks on the code it had just changed, fix what it found within the scope it had touched, and report back before moving on.
Same model, same scaffold, same task, same acceptance tests, same starting instructions. The only variable was whether the agent had a deterministic check in its loop, or just the instructions it started with.
What did the Sigrid guardrails change to the output of Claude Sonnet 4.6?
Adding a real-time quality and security check to the agent’s loop reduced high-risk security findings by roughly 97% and improved maintainability score by 24%, with every guided build outscoring every unguided one.
It also points to a more efficient use of tokens, not a more expensive one, by catching problems before they trigger rounds of regeneration.
Let’s take a deeper look.
Result 1: a 97% drop in high-risk security findings
A 97% drop in high-risk findings is insanely high. It matters because most security breaches start in the code itself. A vulnerability is not bolted on after the fact. It is written in, line by line, the same way every other piece of functionality is.
Most breaches do not start with a brilliant hacker. They start with weak software. Sloppy patterns, missing input validation, an access control check that was never added because nobody asked for it under deadline pressure. Source code is the foundation, and a foundation with a crack in it shouldn’t be considered stable enough to build upon.
According to the State of Software 2026 report, 71% of all code we analyzed last year (so not just AI-generated code) has a low degree of security controls.
This is worrying, as many might think AI will make everything better automatically. In that same report we also found that AI-generated code carries roughly 2X the security risk violations compared to human-written code.
Worth noting here that having a high degree of security controls does not guarantee full security. It just indicates that security considerations have been factored into the design and implementation, making vulnerabilities less likely.
But it wasn’t just the security findings that stood out.
Result 2: Claude Sonnet 4.6 scored 24% better on maintainability
A 24% improvement of the maintainability score, is also huge. It might not sound as cool or important as security findings, but having maintainable code is more important for AI agents and coding assistants than you may think.
Maintainability is best understood as how easily a system can be changed, corrected, or adapted once it exists, what we sometimes call build quality.
Think of a house built with true right angles and solid foundations. Hanging a picture, adding a window, or extending a room is straightforward, because nothing is fighting you. A house built slightly out of whack makes every later job harder, slower, and more expensive, even though the house looked finished on the day it was handed over.
Code works the same way.
Maintainability matters not only for humans, but for agents as well. AI tools and agents work better building on top of already highly maintainable code then working with lower-quality code. Making this improvement well-worth it and for the long run. It also just costs much fewer tokens.
Are we saying you shouldn’t use Claude Sonnet 4.6 or AI?
No. None of this is a story about Claude Sonnet 4.6 being a weak model or AI not powerful.
An agent working from instructions alone, however good those instructions are, has no way to predictably verify its own output against an objective standard mid-task. It can follow the rules it was given. It cannot check, in real time, whether it just broke one.
This is because AI has a limited context-window and takes decisions based on probability, rather than relying on fixed, rigid rules. Which is exactly why having guardrails in place is so important.
Can't you just ask another AI to check the code?
This is a common question I hear a lot. Can’t an agent just ask another AI to check its work? In a sense, yes, but that does not solve the problem.
LLMs, AI coding assistants, and other AI agents can generate and even review code, but they share the same limitation. They reason probabilistically, which makes them fast but not reliably accurate, and a second model checking a first model’s code is still a guess checking a guess. Neither has full context on your system or your architecture.
Here is the simplest way to see it. Ask a frontier model what is wrong, security-wise, with a piece of code, and it compares that snippet against an enormous number of similar patterns it saw during training, then hands back what is best described as a stochastic assessment.
That works well most of the time. It is still not deterministic. It is using statistical models and probability distributions to handle uncertainty, rather than relying on fixed, rigid rules at scale, however convincing it sounds.
In the same SIGNAL podcast episode with Luc, we also spoke about how LLMs are kind rather than honest, because they are trained as helpful assistants rather than truth-seekers and what that means when multiple agents work together.
As I said on that same episode:
“I think what’s sometimes is being underestimated is that when you talk to Chat GPT and you ask it a question, it’s very sweet and very kind, and very forgiving. Imagine that a whole bunch of these LLMs are talking to one another. They’re going to be very much in agreement.”
An MCP server changes what is possible.
It lets the model call out to an external and deterministic source that runs the same analysis and returns the same answer every time, rather than relying on the model’s own best guess.
For agentic workflows this is essential, as an agent is not answering once, it is using tools and carrying out tasks across many steps without a person checking each one.
In a weird way, you can see our platform as being the reliable calculator that you can use to factcheck and guide your developer agents and ensure everything adds up.
AI cost and token efficiency
There is an instinct to assume a check in the loop is overhead. In agentic coding, the bigger cost is usually the opposite: the tokens spent regenerating code that was not right the first time. A deterministic check that catches a problem at the moment it is written tends to reduce that spend.
The economics of agentic coding are unusual. State of Software 2026 cited research that found that agentic coding tasks can consume up to 1,000 times more tokens than a standard code chat.
A thousand times more. That’s a huge difference, and it has everything to do with how agents consume tokens. Because an agent does not answer once. It reads the task, generates code, takes an action, re-reads everything, and often loops back to fix what it just produced, sometimes pulling in other agents that re-read the same context again.
In another SIGNAL episode, we talk about the cost of AI coding, and why developer agents consume more tokens. As I said on that same episode:
“If agents collaborate and they need to talk to each other, they give all contexts. So there a lot of information, just pumped through them continuously.”
When an agent finds and fixes a maintainability or security problem the moment it appears based on a deterministic (predictable) finding, it can resolve the issue while the context is still small and the fix is cheap. The alternative is discovering it ten steps later and regenerating a much larger body of work to address it.
Getting it right the first time leads to fewer (and expensive) correction cycles.
There is a second saving that matters. More capable models are usually a lot more expensive too.
At the time of writing this article, Anthropic just (re)released their latest and newest model; Claude Fable 5. As you can see from the screenshot taken from their pricing page below, the token consumption cost is double compared to earlier models.
So, if the quality and security bar can be enforced by a deterministic guardrail rather than by the model’s own judgment, you no longer need the most expensive frontier model just to feel confident in the output.
A lighter, cheaper model (or open models) paired with a real-time check can reach a standard the same model could not reach alone. The check does the part the model is unreliable at, which means you can route more work to a model that costs a fraction as much for the part it is good at.
What does this mean for teams shipping AI-generated code?
This result is one experiment, on one task, with one model. It is not a universal claim about every coding agent or every codebase, and we are treating it that way.
We would like to see this replicated across more tasks, more models, and larger systems, and we are planning to do more experiments like this one.
What it does show clearly, and what it echoes from Cursor’s FastRender and the agentic C-compiler and human-in-the-loop Openclaw experiments, is that the gap between an agent that writes code and an agent that writes code you can trust is not closed by a better model.
It is closed by what surrounds the model: the checks it runs against, and whether those checks happen in real time or get discovered later.
AI doesn’t necessarily make your code better; it amplifies what’s already there.
So, when you adopt developer agents for software engineering, please ensure they work on a foundation that you can build upon and give them clear rules so that you can amplify the good fast. Instead of accelerating in the dark.
In enterprise organizations, the challenge I often see has little to do with writing new code. The issue is ensuring that that new code can successfully integrate into existing systems that are critical to the business.
If you want to know whether your AI assisted coding is accelerating in the right direction please feel free to reach out.
Guardrails for AI coding assistants and agents
About the author
Werner Heijstek
Werner Heijstek is the Senior Director at Software Improvement Group and host of the SIGNAL podcast, a monthly show where we turn complex IT topics into business clarity.
Frequently asked questions
Why did we test the quality of Claude Sonnet 4.6?
We chose Sonnet 4.6 because it is one of the most widely used coding models in production today.
Anthropic positioned it as the model built for daily use, scaled production, and agentic coding at scale. At the time of this experiment, it was Claude’s flagship daily-driver coding model.
It also scored 79.6% on SWE-bench Verified at launch, within roughly a point of the Opus tier available at the time, which is why it became the daily-driver coding model for most developers rather than a niche choice
What is Sigrid® Guardrails?
Sigrid Guardrails is one of the ways we implemented the MCP standard. It connects Sigrid®, our software portfolio governance platform, directly into an AI coding agent’s workflow, so the agent can call on Sigrid’s analysis while it is generating code, not after.
The specific capability we tested here is the Guardrails MCP: a mandatory quality gate that checks code for maintainability and security issues the moment it is written, and asks the agent to fix what it finds before moving on.
Do quality guardrails make agentic coding more expensive?
Usually the opposite. A real-time check tends to lower total token consumption rather than raise it, because catching a problem early avoids the expensive regeneration cycles that dominate agentic token spend. It can also let a cheaper model meet a quality and security bar it could not reach alone.
Why can I trust Software Improvement Group’s findings?
Software Improvement Group (SIG) has the world’s largest benchmark dataset: 400B+ lines of code, 30,000+ systems, and 300+ technologies. In addition, SIG operates the world’s first and only laboratory accredited under ISO/IEC 17025 for software quality analysis, certified in collaboration with TÜViT.
SIG’s lab meets the strict international standards for testing and calibration laboratories. SIG’s models use standardized, repeatable measurements of source code based on globally recognized ISO/IEC standards, including ISO/IEC 25010 for software quality, ISO/IEC 5055 for code quality, and ISO/IEC 27001 for information security management. In addition, we co-developed ISO/IEC 5338 the new global standard for AI lifecycle management and actively collaborate with NEN on developing and refining ISO and NPR guidance for software and systems engineering, as well as AI and big data standards.
How did Software Improvement Group run this experiment?
We built the same project management system 20 times with Claude Sonnet 4.6 (10 guided runs vs. 10 unguided runs), split into two arms. Both arms used the identical task, scaffold, acceptance tests, and upfront Code Principles instructions. The only variable was the presence of Sigrid®’s Guardrails MCP and its mandatory quality gate in the build loop. Maintainability and security were scored with the SIG maintainability and security models in Sigrid®, the same ISO/IEC 25010-based five-star models applied across SIG’s benchmark of more than 30,000 systems. Security findings map to the OWASP Top 10. The separation between arms was complete: every guided run scored above every unguided run, so the two distributions did not overlap. This is a single experiment on one task and one model. The results are a point-in-time finding, not a universal claim, and replication across more tasks, models, and system sizes is planned.