AI Code Review: Why Loops Without Tests Are Dangerous

In Part 1 we sorted out the three market terms: Context, Harness, Loop Engineering. But Addy Osmani himself warns of a concrete risk: loops without verification keep running, even when the output is wrong. "Whoever writes the loop often no longer understands the problem in detail." This is about the structurally harder side of AI code review: not the tools that check code, but humans reviewing AI-generated code. In project work we know the pattern: AI-generated code looks convincing but fails on edge cases a linter would never find. Why verification is a structural principle, not an afterthought, and what that looks like in practice. That is what this post is about.

What Is AI Code Review? A Brief Framing

AI code review refers to two things: AI tools that automatically review human-written code, and humans reviewing AI-generated code. Both meanings are used synonymously in the market, even though they carry different risks. Whoever fails to separate them builds the wrong kind of verification.

As soon as AI code generation runs at any meaningful scale, classic dev workflows become the bottleneck. They are designed for human reviews, not for the output of a coding agent. Code emerges faster than anyone can review it. The Theory of Constraints calls this the wandering bottleneck: more throughput at one stage cripples the next.

The answer to this bottleneck is a paradigm shift: from human-in-the-loop to human-on-the-loop. Human-in-the-loop means the human is involved in every review cycle. Human-on-the-loop means the human oversees at the process level and only intervenes on escalation. The difference is not convenience but necessity: whoever manually reviews every output creates the next bottleneck themselves.

Tools that automatically review code are known from Greptile, Cloudflare AI Review, GitLab Duo and a growing number of competitors. They take a pull request and deliver comments, risk notes and improvement suggestions. Roborev goes one step earlier: continuous review via Git hook on every commit, not only at the PR. The human review of AI-generated code happens in every team that works with Claude, Cursor or similar tools. It is the less visible but structurally more important variant.

AI code review is not a replacement for classic linters. A linter checks syntax and style rules. AI code review checks meaning: semantics, architecture fit, possible edge cases.

LLM-as-judge has its place. Whether it delivers real verification depends on how it is set up. With full context of the coding loop it is white box: it shares the same assumptions. Isolated, focused only on output and user scenario, it can be an independent reviewer.

The confusion of terms leads to false expectations. Whoever sells "we introduced AI code review" and means a tool has not addressed the human reviewer problem. Whoever says "AI code review" and means the human check does not have a tool stack that keeps up with the speed. Both meanings need different answers. This post focuses on the second, because it is the larger structural problem.

The Three Risks From Osmani's Warning in Practice

Addy Osmani warns in his original blog post on loop engineering of three risks, the honest flip side of his thesis, not a footnote. We at codecentric have been working with autonomous loops in client projects for months and can confirm all three.

Risk 1: Insufficient Verification of Unattended Loops

Loops that write their own tests and run against them are blind to edge cases beyond the spec. That sounds obvious until you see it in a project. In one client project an autonomously running agent deleted its own tests because they got in the way during refactoring. From its perspective this was logical: the goal was "all tests green", the path was open. From our perspective it was the symptom of a structural problem. When the agent controls verification itself, it is no longer verification but self-confirmation.

This is the same logic as with a holdout set in ML. A model that can see its own validation set validates itself. The validation loop needs an instance that was not involved in the generation.

Wes McKinney, creator of pandas, Apache Arrow and Ibis, and himself deep in agentic workflows, puts it succinctly: "If all of your code isn't being automatically reviewed by adversarial agents, you've essentially got tons of bugs lurking that you can't possibly find through your own human QA." (Source).

Roborev, the open-source tool built by McKinney himself, implements exactly that: automatic review on every commit via Git hook, before errors stack up.

Adversarial agents are the automated form of this: independent agents explicitly designed to refute the output of the coding agent. Isolated Specification Tests are the structural foundation for it: a test layer that defines the target behaviour from the user's perspective and to which the agent has no access, neither read nor write. We have described the concept in a dedicated post.

Risk 2: Loss of Understanding Through Speed

When a loop generates 200 lines of code in five minutes, the human reviewer can no longer grasp the code semantically. They see: compiles, tests green, looks plausible. They do not see whether the chosen architecture fits the surrounding system, whether the data model assumption holds, whether the library version is still supported.

AI-generated code is particularly tricky here: stylistically consistent, idiomatic, well named. The errors do not sit on the surface but in architecture assumptions and edge cases underneath. Review at that level costs more cognitively than with hand-written code, not less.

Joni Klippert, CEO of the security startup StackHawk, reports on a financial services company that jumped from 25,000 to 250,000 lines of code per month after introducing AI coding tools and built up a review backlog of one million lines. Speed without reviewer capacity shifts the bottleneck; it does not solve it.

Targeted slowdown at critical points helps. Architecture decisions, data model changes and security-relevant sections need explicit senior review gates. Not every PR has to go slow, but the structurally critical ones do.

Risk 3: Cognitive Capitulation

The third risk is the quietest. When every loop output "mostly fits", the brain switches off. In organisational research this pattern is called Normalization of Deviance: what initially passed as exception becomes the norm, until no one asks why anymore. Add to that the IKEA effect: self-generated output is systematically rated as more valuable. That applies to the team that "built the output themselves".

In client teams we observe the same effect after four to six weeks of autonomous loop work: "looks good" becomes the dominant assessment, "check again" the rarely voiced demand. Not a discipline problem. Generation is faster than understanding. The brain adapts.

The answer is structural, not cultural. Cultural verification does not scale over weeks. Structural does: tests that must run independently; governance loops no human can skip. The discussion "let's take reviews seriously!" is not enough.

The Trilemma: Three Paths, None of Them Good

Without structural verification, three options remain, and none is good:

• Human Oversight: does not scale when loops get faster than reviewers
• Exhaustive Audits: too slow for continuous delivery
• Vibe Accept: the path no team consciously chooses but many take in practice

The four measures below are not a best practice list. They are the way out of this trilemma. Isolated Specification Tests take over where humans can no longer keep up. Review gates deliberately slow down the structurally critical points without slowing down all delivery. And whoever budgets the verification effort treats it as what it is: investment protection, not overhead.

White Box vs. Black Box: The Verification Table

Verification has two perspectives: white box with access to the code, black box independent from outside. Both are needed because they find different classes of errors.

Concern	White Box (with code access)	Black Box (independent)
Security	Static Analysis, Dependency Scanning	Penetration Testing from outside
Functionality	Unit Tests, Integration Tests	E2E tests from user perspective
Performance	Profiling, Benchmarks	Load Testing in Staging
Compliance	Automated Compliance Checks	Independent acceptance, Audit

The temptation to rely on white-box verification is strong: faster, automatable, runs in CI. But it has the same blind spot as the coding loop itself. It only sees what the spec says. Black-box verification is structurally necessary because it tests from the perspective the coding loop does not have: that of the user, the attacker, the audit.

What You Can Do Now

1. Introduce Isolated Specification Tests: tests the agent is not allowed to modify, e.g. in a separate repository or via .claudeignore and agent permissions that revoke the coding agent's access to the test suite.
2. Senior review gate for architecture and data model changes
3. Black-box verification as a mandatory stage in CI/CD
4. Budget the verification effort explicitly

Whoever pays the Verification Tax lands at the next bottleneck: no longer implementation, but selection. What do we even build? Without strategic selection, faster loops produce more features, not better products. Verification solves the problem of correctness. The problem of relevance is the topic of Part 3.

FAQ

What is the difference between AI code review and classic code review? Classic review checks code that humans wrote. AI code review refers to two things: AI tools that automatically review code (Greptile, GitLab Duo) and humans reviewing AI-generated code. The second variant is the structurally harder one: AI-generated code is stylistically convincing, the errors sit in architecture assumptions and edge cases underneath.

Are unit tests enough for AI-generated code? No. The agent often writes its tests itself, against its own spec. What it models incorrectly does not show up in its tests. On top of that, agents delete tests that get in the way during refactoring. Black-box tests the agent cannot modify are mandatory.

When does human review turn into Vibe Accept? When the reviewer no longer semantically understands what they are reviewing. AI code is well named and idiomatic, the errors sit underneath. Whoever only checks "compiles, tests green, looks plausible" is no longer really reviewing. That is not a discipline problem: generation is faster than understanding, the brain adapts.

What are Isolated Specification Tests? Tests the AI agent cannot see or modify, in a separate repository or via permissions that revoke access. They define the target behaviour from the user's perspective. The target behaviour is defined externally, the agent cannot clear it away.

Are tools like Greptile, Cloudflare AI Review and GitLab Duo the solution? Building blocks, not a replacement for structural decisions. These tools automatically review pull requests and deliver risk notes. They help little if the coding loop itself does not enforce verification, that is, if the agent has access to its own tests or if no independent review gate exists.

Further Reading

• Birgitta Böckeler — "The role of developer skills in agentic coding" (martinfowler.com) — why steering and correcting are the core competences in working with coding agents
• Roborev — open-source tool by Wes McKinney for continuous AI code review via Git hook.
• GitLab Duo Agent Platform — GitLab's AI platform for the entire SDLC, including specialised agents for code review, security analysis and agentic workflows.
• codecentric: No Cheating: Isolated Specification Testing with Claude Code
• codecentric: Part 1 of the series — Loop, Harness, Context Engineering: what means what?

---

In Part 3 we look critically: where loops fail in reality — and what Boris Cherny would say if reality did not run as smoothly as his tweet.