Opinion
From human in the loop to human in control: what the five levels of AI-driven development mean for a security company
I have a confession to make. In our development team, almost nobody reads code line by line any more. A year ago I would have called that negligent, especially for a company that builds security software. Today I am convinced of the opposite: the tools we use check our code more carefully than we ever did ourselves.
Two recent reads helped me put words to what we are experiencing. The first is Dan Shapiro’s blog The Five Levels, published in January 2026, which describes five levels of AI-driven software development, from “spicy autocomplete” to the “dark factory”. The second is Ethan Mollick’s new book Co-Existence, published in October 2026, about living and working alongside machines that are sometimes smarter than we are. Reading them side by side, I realised that our company has reached level four and is slowly growing towards level five. More importantly, I realised that one of the most persistent assumptions in our industry deserves to be revisited: the idea that a human must always be in the loop.
What are the five levels of AI-driven software development?
Shapiro borrows his framework from the levels of driving automation. At level zero, every character is typed by a human. At level one, you hand off small, discrete tasks such as writing a unit test. At level two, you pair with AI like a colleague; according to Shapiro, this is where most self-declared “AI-native” developers live. At level three, the AI writes the code and the developer becomes a reviewer who spends the day reading diffs. Shapiro observes that almost everyone tops out there.
Level four is different. The developer is no longer a developer, nor a development manager, but something closer to a product manager. You write the specification, you argue with it, you build the skills your agents use, you review plans, and then you walk away and check later whether the tests pass. Level five is what Shapiro calls the dark factory: a black box that turns specifications into software, named after the robot factories where humans are neither needed nor welcome.
Our engineers now spend their time on specifications, skills and test design rather than on reading diffs. That is level four. We are not at level five, because humans still decide what gets built, how it is tested and what is allowed into production. As I will explain, I am not sure we ever want the lights to go out completely.
Is it safe to let AI write security software?
It is a fair question, and one I asked myself many times. The evidence that unchecked AI-generated code is risky is real. A research note from the Cloud Security Alliance cites an analysis by CodeRabbit of real-world pull requests, which found that AI-generated code contained 2.74 times more security issues than code written by humans. A large-scale academic comparison published in 2025 reached a similar conclusion: AI-generated code tends to be simpler, but it is more likely to contain high-severity vulnerabilities.
Some companies have gone much further than we have. In February 2026, the AI team at StrongDM, a company that builds access management and security software, described its “Software Factory”, governed by two rules: code must not be written by humans, and code must not be reviewed by humans. Simon Willison, who visited the team, remarked that security software is the last thing you would expect to be built this way. Researchers at Stanford Law School raised a sharper objection: when the same class of technology both writes the code and judges whether it works, you risk a circular system that grades its own homework.
I take both points seriously. But read carefully, they point to the same conclusion. The risk does not lie in AI writing the code. It lies in the absence of serious, independent verification. That is where the real work is.
Can AI review code better than humans?
For a long time we assumed that humans had to check the work of AI. What we have discovered in practice is that AI is far better at checking the work of humans, and of other AI.
This is not wishful thinking. When OpenSSL, one of the most scrutinised codebases in the world, published twelve new vulnerabilities in January 2026, all twelve had been discovered by the AI system of the security start-up AISLE, according to the company itself. In its May 2026 paper The Bugpocalypse Threshold, the Cloud Security Alliance describes how AI tools can now review millions of lines of code in hours rather than weeks. It also notes that Google’s Big Sleep agent became the first documented AI agent to find a previously unknown, exploitable flaw in widely deployed software, the SQLite database, without human direction.
Let us be honest about human review. We get tired. We skim. We trust a colleague whose work has always been fine. We approve the large pull request on a Friday afternoon because it looks reasonable. A well-instructed machine reviewer does none of those things. Moving away from manual review felt uncomfortable at first, almost like letting go of the steering wheel. That changed when we saw, again and again, that the tools caught what we would have missed.
Regular readers may notice a shift. In October I wrote that every change at our company is reviewed by a person who understands it before it reaches production. In one sense that still holds: a human decides what goes into production. What has changed in the weeks since is who, or what, does the reading. The line-by-line review has moved to our tools, because they turned out to be more thorough than we were.
This does not mean that AI-written code is safe by default. The research above makes clear that it is not. It means that verification by AI, if designed well and layered properly, is more thorough than verification by humans alone.
Where does control actually live?
“Human in the loop” means that a human checks every step. “Human in control” means that humans design the system that does the checking, and decide what is good enough. Think of an airline pilot. The autopilot flies most of the journey, yet nobody considers the pilot redundant. The pilot is in command, not in the loop of every adjustment.
At Guardian360, control starts before a single line of code is written. Our specifications describe not only use cases but also abuse cases: how could this feature be misused, and by whom? We have instructed our AI tools to be critical and explicitly not to behave like a helpful intern. A helpful intern wants to please you, and an agent that wants to please you will write code that passes the test rather than code that is right. Behind that sit the layers you would expect in any mature environment: linting, hardening, segmentation and role-based access control, together with functional, integration and unit tests. Several models review the code from different angles, so that no single model marks its own work.
One choice matters more than it may appear. We run an open-weight model ourselves, in our own server cluster in the Netherlands, with our own skills, capabilities and harness, and a fixed test that every model has to pass before we trust it. When you own the harness, you decide what agents can see, touch and do. Control is not only a matter of oversight; it is also a matter of ownership.
What happens when code is no longer the bottleneck?
In October I described how we rebuilt our platform in roughly three months, work that previously took more than three years. The constraint has moved from building to implementation: keeping partners informed and keeping the product simple. When building costs almost nothing, the discipline we need is not the discipline to deliver, but the discipline to say no. Speed versus security remains my biggest concern, which is exactly why we invest in verification rather than in more hands on keyboards.
What happens to developers when coding is no longer the job?
I have written before about the human cost of AI-assisted development and about what it does to the undercurrent of an organisation. Moving from level three to level four adds a twist I did not expect. Research I cited earlier found that overseeing AI is the most mentally taxing form of AI use, and reading diffs all day is exactly that. At level four, engineers stop performing the checks and start designing them: writing specifications, defining abuse cases and deciding what “good” looks like. That is more demanding intellectually, but it may well be less draining.
Shapiro jokes that at level four you have become what you used to loathe, a project manager. In our team it feels more like becoming an architect of trust. Not everyone embraces that immediately, and as a leader I think it is important to say so out loud. For some, letting go of the code feels like a loss of craftsmanship, and the culture of a team is shaped by what people spend their days on.
Level five, with the lights on
Shapiro’s dark factory is a place where humans are neither needed nor welcome. That is not where we are heading. We are growing towards level five in the sense that more and more of the process runs autonomously, from specification to working software. But the lights stay on.
Not because we distrust the machines, but because responsibility cannot be delegated to a factory. Our partners and their customers do not trust us because a human has read every line of code. They trust us because humans remain accountable for the system that produces it, and for the decisions about what it is allowed to do.
So I would like to leave you with two questions. Where is your development team on Shapiro’s scale? And when something goes wrong, can you say who was in control, rather than only who was in the loop?
Sources
- Dan Shapiro, “The Five Levels: from Spicy Autocomplete to the Dark Factory” (23 January 2026), https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/
- Ethan Mollick, “Co-Existence: The Next Phase of AI” (Portfolio / Penguin Random House, October 2026), https://co-existence.ai/
- Simon Willison, “How StrongDM’s AI team build serious software without even looking at the code” (7 February 2026), https://simonwillison.net/2026/feb/7/software-factory/
- Stanford Law School, “Built by Agents, Tested by Agents, Trusted by Whom?” (2026), https://law.stanford.edu/?p=557739
- Cloud Security Alliance, “Vibe Coding Security Crisis: Credential Sprawl and SDLC Debt” (research note, 2026), https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-generated-code-security-vibe-coding-202/
- “Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity” (arXiv, 2025), https://arxiv.org/pdf/2508.21634
- Cloud Security Alliance, “The Bugpocalypse Threshold” (21 May 2026), https://labs.cloudsecurityalliance.org/research/ai-accelerated-vuln-discovery-patch-capacity-whitepaper-v1-0/
- AISLE, “AI found 12 of 12 OpenSSL zero-days” (January 2026), https://www.greaterwrong.com/posts/7aJwgbMEiKq5egQbd/ai-found-12-of-12-openssl-zero-days-while-curl-cancelled-its
- Guardian360, “We now build in three months what took us three years” (12 October 2026), https://guardian360.net/blog/we-now-build-in-three-months-what-took-us-three-years/
- Guardian360, “AI took away the typing, not the thinking” (19 October 2026), https://guardian360.net/blog/ai-took-away-the-typing-not-the-thinking/
- Guardian360, “AI agents need no social maintenance. Your organisation does” (25 October 2026), https://guardian360.net/blog/ai-agents-need-no-social-maintenance/
- Guardian360, “We tested our self-hosted LLMs and accidentally built a game” (1 October 2026), https://guardian360.net/blog/we-tested-our-self-hosted-llms-and-accidentally-built-a-game/