Going From Vibe‑Coding Agents to Production‑Grade AI Security

You can ship your first MCP server or "agent" in a weekend. You can also accidentally ship a very helpful little robot with root on your company.

Ask me how I know.

I walked into a codebase once where the "genius dev" shipped a flashy AI thing, rode the internal hype cycle for a few months, and then bounced to a bigger logo. The product was live. Customers were real. The security debt was now everyone else's problem, and the AI part was the least of it.

This post is for that Monday.

The Monday after the demo

The pattern is predictable:

• The agent works. It calls tools. It "reasons." It books meetings, queries data, generates reports.
• There's a .env permanently "temporary" in the repo.
• The tool list includes something like run_sql(query: str).
• Your logs read like a greatest-hits album of secrets and customer data.

If you're anywhere near healthcare, finance, or EU users, the temperature goes up fast. Now you're not just worried about your logs. You're worried about some future audit where you have to explain why "full prompt + tool output + PHI" lives in a random log bucket that four vendors can access.

At a previous SaaS job, we were selling "AI automation" for financial workflows. Under the hood, most of it was boring: spreadsheet column mapping, data cleaning, CSVs in, CSVs out. It worked, but the scary part was never the model. It was where the data went, who could touch it, and how much glue code quietly had prod credentials wired in.

Agents are the same. The model is the flashy part. The plumbing is the blast radius.

The triage checklist (before you add one more feature)

1) Rotate keys and get them out of the repo

If an API key ever touched your code or config in plain text, treat it as compromised. Rotate it. Yes, even if the repo is "private." Your SCM is not a vault.

Do this:

• Rotate vendor keys (OpenAI, Anthropic, AWS, database creds, all of them).
• Scrub secrets from git history, not just the last commit.
• Move secrets into an actual secret manager (AWS Secrets Manager, GCP Secret Manager, Vault).

Baseline rule: no long-lived keys on laptops, in repos, or sprayed through CI logs.

2) Put a proxy between the agent and the world

Most teams wire agents straight to model endpoints and tools and call it a day. That's how you wake up with an "AI assistant" that can talk to prod like an unsupervised admin.

Drop an HTTP proxy in front of the LLM and tools. One choke point, one place to enforce policy.

Minimum features:

• Rate limiting per user, per agent, maybe per tool.
• Structured request/response logging instead of print() spam.
• Redaction (regex-based is still better than full-text confessionals).
• Allow/deny lists for tools so not every agent can call everything.

Something like this is enough to stop reading tea leaves in CloudWatch later:

import json, time

def log_event(event_type, user_id, agent_id, tool_name=None, meta=None):
    payload = {
        "ts": time.time(),
        "event": event_type,
        "user_id": user_id,
        "agent_id": agent_id,
        "tool": tool_name,
        "meta": meta or {},
    }
    print(json.dumps(payload))

Now at least you can answer "who called what, when, and which tool" without doing archaeology across 15 different log streams.

3) Split tools and shrink scopes

If your agent can dump the prod database, assume one day it will. Not because it turned evil, but because you gave it a tool called get_customer_context() that happily returns everything.

Make tools boring and narrow:

• get_invoice_status(invoice_id)
• Not query_finance_db(sql)

Then scope credentials per tool:

• Separate read vs write.
• Separate staging vs prod.
• Separate "support agent" from anything admin-shaped.

Agents work a lot like small models: narrow tools with tight permissions behave better than giant god-mode utilities. If you're wiring MCP servers or agent frameworks, treat each integration like its own little blast chamber: its own keys and limits, not one shared root password.

4) Treat logs like toxic waste

If you log everything "for debugging," you're not instrumented; you're running a long-term data leak.

Basic rules:

• Do not log raw prompts by default.
• Do not log full tool outputs, especially from anything touching customer data.
• Hash or tokenize identifiers where you can.
• Set retention limits and stick to them.
• Lock log access down like production data, because that's what it is.

Teams often forget how many people have indirect access to logs: support tools, BI dashboards, contractors, random "observability" vendors. If your logs contain PHI or payment data, you didn't build observability. You built evidence.

5) Run a tiny DPIA / HIPAA-style risk check

You don't need a 40-page Word doc. You do need a clear picture of where the risk sits.

Ask and actually write down:

• What sensitive data flows through this system?
• Where does it land: logs, caches, backups?
• Who can access it, inside and outside the company?
• Which vendors process or store it?
• If something goes wrong, what's the blast radius and who gets hurt first?

That's basically a DPIA-lite: a short, direct risk assessment for a specific data flow. If you can't answer those questions quickly, you're not moving fast. You're driving with your eyes closed.

The next 48 hours

If the AI thing is already in prod and your stomach drops when you think about the logs, here's the short version:

• [ ] Rotate keys and scrub git history.
• [ ] Put a proxy in front of the model and tools.
• [ ] Kill any "run arbitrary SQL" style tool in prod.
• [ ] Lock each tool to least-privilege credentials.
• [ ] Turn off prompt/tool output logging by default.
• [ ] Write a one-page data flow and risk note (your DPIA-lite).

Shipping the demo is the easy part. The grown-up work starts on the Monday after the applause.

- Sethers