AI in CI/CD: Let the Bot Write, But Make It Earn the Deploy

AI “helpers” are great right up until they quietly rewire your release pipeline and punt prod into a ditch.

I watched one do it with a YAML tweak so small it barely showed up in the diff. Actually, I didn’t watch it. I rubber‑stamped it. Everyone did.

Three-line diff. Checks all green. I skimmed it, thought “yeah, fine,” and hit approve. That was the bug.

The change shuffled when a deploy job ran, so it started firing on branches it had no business touching. A “safe” script ran twice. The second run wasn’t idempotent. We got a half‑rolled deploy and a stack of alerts that all said “something is wrong” and none that said “go look at this one line of YAML.” sethserver

This is the part of CI/CD that still makes me nervous. You can vibe‑check code all day, but once the pipeline moves, the blast radius is real. sethserver

Where AI quietly creeps in

AI doesn’t start by rewriting your core app logic. It shows up in the boring corners: PR blurbs, config “tidying,” little refactors nobody wants to think about. sethserver

• PR generation: “I fixed the lint errors and updated the Dockerfile.”
• Config refactors: “I reorganized your GitHub Actions for readability.”
• Test suggestions: “I added unit tests for the edge cases.”
• Migrations: “I wrote the Alembic migration for that new column.”

All of that sounds helpful. None of it sounds dangerous.

Then you remember YAML is basically a footgun with whitespace, and migrations are SQL booby traps running on a schedule.

What breaks even with green tests

Treat AI like a junior programmer who never gets tired, never feels awkward, and will confidently propose the worst possible idea with a straight face.

Here’s what I’ve run into:

1. Subtle misconfig

2. Wrong paths: filter, wrong if: condition, wrong environment on a job.

3. Suddenly your deploy runs on PRs, or a staging job gets wired up with production secrets.

4. Security regressions

5. AI “simplifies” IAM, loosens a security group, or trims a step that was doing signing or verification.

6. You end up with a pipeline that flies through the checks and leaves the door propped open.

7. Non‑idempotent scripts

8. Anything with verbs like create, apply, seed, migrate, sync is guilty until proved safe.

9. If a script can run twice, assume one day the pipeline will oblige.

10. Infra drift

11. AI edits Terraform modules or Helm charts without any sense of your weird internal constraints.

12. Everything “works” until the next deploy wedges itself into some corner of your cloud and refuses to move.

I used to work somewhere that sold “AI automation.” Under the hood it was spreadsheet mapping and a human in another time zone. The illusion held until you asked it to explain a decision. Same thing here: green checks only mean you passed the tests you actually wrote, not the ones you should have.

AI proposes, you own the blast radius

AI doesn’t deploy. It drafts. You’re the one putting your name on the outage.

So you lock down the obvious basics:

• Protected branches.
• Required checks that can’t be bypassed “just this once.”
• Human review from someone who actually understands the system.
• Ownership rules that don’t dissolve because a VP wants something “by end of day.”

If your bot can merge its own PRs, you didn’t build automation, you built an outage generator with a pretty UX.

Files that always need real eyes

There are parts of a repo where AI can help, but it shouldn’t be the last set of eyes:

• infra/, terraform/, helm/
• .github/workflows/ and other pipeline configs
• IAM and permissions
• Anything touching secrets
• Database migrations

Make the rule boring and strict: some changes need senior approval, period. Not “strongly recommended.” Not “if you have time.” A gate.

Tests that actually catch pipeline damage

Unit tests are great for catching logic bugs. They’ll happily pass while your deploy workflow points straight at production instead of staging.

You want gates that exercise the way the system is glued together:

• Integration tests.
• Smoke tests in staging.
• Migration dry‑runs.
• Simple “does it start and respond” checks.

Policy checks that bite

You can use Open Policy Agent, homegrown scripts, or whatever lint‑like thing fits your stack. The tool matters less than the habit of blocking obviously bad ideas.

Easy wins:

• No security groups open to 0.0.0.0/0 by default.
• No new GitHub Actions permissions without explicit review.
• No deploy job changes without sign‑off from whoever owns releases.

Here’s a small guardrail idea in pseudo‑code:

def blocks_ai_pr(diff, labels):
    if "ai-authored" not in labels:
        return False
    risky_paths = ("infra/", ".github/workflows/", "migrations/")
    return any(path.startswith(risky_paths) for path in diff.changed_files)

Simple rules like that save you from very clever, very avoidable outages.

Treat AI PRs like they’re spicy

Label AI PRs automatically, then make them earn their way through:

• Config linting.
• Security scanning.
• Static analysis.
• Migration dry‑runs.
• Plan steps for infra (terraform plan, helm template, etc.).

Canary PRs help a lot. Let the AI propose the change, but stick it behind a feature flag or roll it out to a tiny slice first. If it breaks, it breaks small and noisy instead of big and mysterious.

Start boring on purpose

Don’t kick things off by letting an AI remodel your infra. Start where mistakes are cheap.

1. Docs edits.
2. Test generation.
3. Small refactors in app code.
4. Low‑risk config files.
5. Then infra and migrations, but only with real review and extra gates.

Also be clear on ownership. When an AI‑assisted change goes sideways, who’s on call, who hits revert, who explains what happened? If your answer is “the bot,” you’re not ready.

If you’re still shipping vibe‑coded changes you don’t understand, you’re just rolling dice with a nicer UI. The jump from vibe‑coding to production‑grade AI security is mostly process: labeling, gates, and humans who feel responsible for what ships.

Quick checklist (10 yes/no)

Run through this and count your “yes” answers:

1. Does your AI have write access to main?
2. Can it merge its own PRs?
3. Are AI PRs labeled automatically?
4. Do protected branches require human review?
5. Are deploy workflows locked down?
6. Are secrets blocked from PR jobs?
7. Do you run smoke tests in staging?
8. Do migrations get dry‑run or plan checks?
9. Do you have any policy gates on infra and pipeline changes?
10. Can you revert a bad deploy in one command?

If you answered yes on #1 or #2, stop and fix that before touching anything else. The AI can help you write code; you’re the one who has to live with what it ships.

-Sethers