Skip to content
Blog

Don't Trust an Agent That Says Done

Oct 1, 2026 4 min read ai, engineering, testing

Coding agents grade their own homework. kanman doesn't let them: every change needs your own CI and a reviewer that did not write it, which reruns the bug before and after the change. The machine that merges checks both.

“All tests pass. The feature is complete.”

Every coding agent says this. Most of the time it’s even true, in the narrowest sense. The tests pass. The tests the agent wrote. For the code the agent wrote. Based on the agent’s reading of a ticket that said “make the export faster”.

That’s not proof. That’s a student grading their own exam and handing you the score.

Where “done” goes wrong

I ran an AI team on a real product for ten weeks before I rebuilt kanman around what I learned. The failures that hurt were never syntax errors. CI catches those.

The ones that hurt looked like this:

  • The agent mocked the exact thing the test was supposed to check. Green run, nothing tested.
  • The agent “fixed” a failing test by changing the assertion.
  • The feature worked on the agent’s machine, with state left over from three earlier attempts, and nowhere else.
  • The PR did what the ticket said, and not what the person who wrote the ticket meant.

Each one passed review at least once, because reviewers trust green checks. Of course they do. That’s what green checks are for.

A second pair of eyes that runs things

So kanman doesn’t ask the agent whether it’s done. Two things decide, and neither of them wrote the code.

1. Your CI. The pipelines your colleagues’ changes go through, nothing invented for the occasion. If a job is red because of the change, the change goes back to work. If a job is red on main too, for the same reason, kanman names the job and the cause instead of quietly ignoring it.

2. A reviewer that didn’t write it. A separate session, never the one that wrote the change, gets the merge request. It checks out the exact commit, reads the full diff against the story, and looks for exactly the failures above: a mocked target, a loosened assertion, a skipped test.

3. It runs the proof itself. The reviewer reproduces the original bug, or runs the story’s acceptance check, on main first. Then it runs the same check on the change, which now has to pass, and runs the tests. When a reproduction isn’t possible, say it needs a production account kanman doesn’t have, it says so and why. I’d rather tell you that than dress up a green CI badge as proof.

4. The rule lives where the merge happens. The coding agent has no token to merge with. kanman’s host merges only when CI is green at the latest commit and the reviewer approved that exact commit with no blocking finding. A new commit after the review means a new review. Your protected branches and approval rules apply on top.

What the reviewer proved goes into the merge request’s description: what it ran, before and after, and its findings, each on the line it is about. Your reviewer reads evidence instead of reconstructing it.

“Our repo has thin tests”

Most repos I see do. That’s fine.

You don’t need a special test setup to start. kanman uses the CI you already have, and the reviewer’s before-and-after run covers the behaviour your tests don’t. Where your CI is thin, that shows up in the reviews, and kanman can propose the missing tests as ordinary stories.

Why this is the part that matters

There’s a reason this sits in the middle of kanman and not in a settings page.

An AI teammate that ships unverified work doesn’t save your team time. It moves the work from writing code to doubting code. Your senior people end up reviewing every line twice, because they can’t tell which PRs are real.

An AI teammate that shows evidence changes the review. You stop asking “does this work?” and start asking “is this what we wanted?”. That second question is the one humans are good at.

Agents that prove their work beat agents that demo. If you want the details, how kanman checks its work is documented step by step.

Want an AI teammate that proves its work before it asks for your review? kanman - Local free, Starter €99 and Team €249 per AI team per month, 14-day trial without a card.
Marco Kerwitz
Author

Marco Kerwitz

Founder of kanman.ai

Meet kanman

kanman is an AI teammate for engineering teams. It takes requirements, writes the stories, ships the code through Claude Code or Codex and proves each change against your acceptance criteria.

  • Works inside your Jira, GitHub or GitLab.
  • Your policy decides what it may touch, spend and merge.
  • Every change passes your CI and an independent review before it merges.
Start the 14-day trial

Local free, Starter €99 and Team €249 per AI team per month, 14-day trial without a card.