People love to say "we have a human review step." I'm guilty of this sin. They ship a UI that turns the human into a rubber stamp. The person clicks Approve, nobody reads anything, and suddenly the company has "human-in-the-loop" on a slide deck.
That's not a loop. That's a checkbox.
I've watched this flop in a few flavors. One of my favorites was a SaaS company that swore they used AI for boring financial work. A few weeks in, I noticed support and services were oddly protective. Simple questions got rehearsed answers. At one point I was literally told, "come to our office and watch us work." Code review later: the "AI" was mostly spreadsheet column mapping and data formatting. Excel cosplay with a GPU budget. The "review" step was a human clicking through a blob of output they didn't understand, under time pressure, with zero context. They predictably approved junk. Then everyone blamed "the model."
Why your review step doesn’t work
Humans happily approve things they don't understand. Especially when:
- The UX is a wall of text
- The reviewer has no mental model of how the system works
- The cost of slowing down feels higher than the cost of being wrong
- The org is thrashing: low trust, lots of deflection, rotating “saviors”
If approvals feel rigid or opaque, trust but verify. Ask, "What's our runway?" for the business. Ask, "What's our rollback plan?" for the system. Same instinct, different layers.
When humans actually need to be in the loop
Some actions are too spicy to run end-to-end without a real review:
- Money movement (refunds, payouts, invoice tweaks)
- Access changes (permissions, account recovery, admin grants)
- Bulk updates (mass edits, deletions, migrations)
- Legal/compliance (policy decisions, regulated workflows)
- Ambiguous model outputs (low confidence, conflicting signals)
If the blast radius is big, the human can't be theater. They need tools that make it hard to sleepwalk into a catastrophe.
Design review surfaces humans can actually use
The fastest way to kill a review step is to dump raw output in a box and call it a day. Don’t do that.
Show diffs, not blobs
If the model edits text, show before/after with a diff.
If it edits structured data, show field-level changes.
- refund_amount: 0
+ refund_amount: 49.99
- refund_reason: null
+ refund_reason: "Duplicate charge"
Humans can scan diffs. Humans cannot meaningfully review a wall of JSON between Slack pings.
Summarize impact in plain language
Put the consequence at the top:
- "This will refund 1,274 users for a total of $63,492.18."
- "This will remove admin access from 12 accounts."
- "This will delete 8,201 records (estimated restore time: 3 hours)."
If your UI can't say what’s about to happen, your system doesn't understand what it’s doing. That’s the real bug.
Put risk where the reviewer will see it
Confidence scores are cute, but you need signals that match real risk:
- anomaly flags ("refund amount 10x user's average")
- policy violations ("refund reason missing")
- novelty ("first time this merchant has requested bulk refunds")
Make the risky bits loud. Reviewers shouldn't have to play Where’s Waldo with the danger.
Friction with intent
Friction is not the enemy. Random friction is.
Use more friction for higher-risk actions:
- Low-risk: one-click approve, clear summary, done
- High-risk: forced checklist, second approver, "type this phrase to confirm" step
Use progressive disclosure:
- Start with summary + key diffs
- Let the reviewer drill into logs, raw model output, and evidence only if needed
People want to move fast. Let them. Just don’t let them bulldoze through something dangerous without noticing.
Guardrails on the human side
If your main button says Approve All, you built a rubber stamp with extra steps.
Guardrails that actually help:
- Batching with limits: approve 20 at a time, not 2,000
- Timeouts: pending approvals should expire
- Decision fatigue controls: rotate reviewers, cap daily volume
- Playbooks: tiny checklists like "If X, check Y," not a 40-page wiki nobody reads
Close the feedback loop
A real loop learns.
Require Reject with reason. Even a dropdown works:
- wrong customer
- wrong amount
- missing evidence
- policy conflict
- low confidence
Then track:
- override rate (how often humans disagree with the model)
- post-approval incidents
- reviewer agreement (two humans, same case, same choice?)
If you can't answer "how often do humans catch real problems?" you don't have a loop. You have theater with a legal disclaimer.
A simple example flow
- Model proposes refunds.
- UI shows: count, total dollars, top anomalies, and diffs per user.
- Reviewer approves low-risk cases quickly.
- Risky cases require a second approver and a typed confirmation phrase.
- Rejections require a reason.
- Reasons feed evaluation, prompts, and tooling changes.
That’s human-in-the-loop. Not human-as-liability-shield.
The same principles apply when AI generates code in CI/CD pipelines: you need real review gates that catch dangerous changes before they ship, not a checkbox that lets broken YAML slide through because everybody’s in a rush.
-Sethers
