sethserver.com Subscribe
A yellow and orange robot with a boxy design holds a tablet displaying colorful analytics data including numbers (136, 61, 32, 68), percentages, charts, and an upward trending arrow. The robot is surrounded by floating numbers (1, 3, 6, 7) in pink and blue, along with geometric shapes like stars, circles, triangles, and squares on a mint green background.

Vanity Metrics Look Better When AI Generates Them

By Seth Black • Updated: September 17, 2026

Startups · 6 min read

Like this kind of writing? Get one email a week with notes on startups, AI, and the occasional strong opinion about Python: subscribe to the newsletter.

AI can generate a lot of stuff.

Landing pages. Cold emails. "Personalized" outreach that reads like a fortune cookie. Product tours. Help docs. Even little charts that look like you're doing science.

And yes, it can generate signups too. Aim it at the top of the funnel, tell it to spray output everywhere, and it will.

I ran into the human version of this problem years before AI could do any of it, at a SaaS company that claimed to use AI to automate mundane, error-prone financial work. A few weeks in, I started asking basic questions. Where's the model? Where's the training data? Where are the evals? The support and services team, based in South Asia, didn't send documentation. They told me to come to their office and watch them work.

Not a joke. That was the actual answer to "where's the model."

So I did a code review instead of a plane ticket. The "AI" was mapping spreadsheet columns and reformatting data. That's it. Excel cosplay wearing a machine learning costume to the investor meeting. And underneath that, there was a second fight nobody put in the pitch deck: an internal "lowest cost" battle, our own support team against a batch of external contractors, both quietly serving the same customers on the side. Everybody racing to be the cheapest labor attached to a product that was mostly labor already. The dashboard still looked great through all of it. The story still sold. The numbers still moved. Just not for the reasons anyone in the board meeting wanted to admit out loud.

Here's the part that actually stuck with me: humans beat computers when humans are cheaper. Every time somebody tells you a workflow is "AI-powered," ask what it costs to run versus what it costs to pay a person in a lower-wage market to do the same task by hand. If the human is cheaper, the human is doing it, and the model is a slide.

AI makes the newer version of that pattern easier, and cheaper to fake at scale. Output gets cheap, so it's easy to confuse activity with progress.

The chart got prettier. The business didn't.

Vanity metrics love volume:

AI can pump all of those. You can ship three new onboarding flows in a week and feel like a machine.

But if your activation rate stays flat, you didn't improve onboarding. You improved the number of onboarding flows.

I still like Lean Startup's "innovation accounting" framing. Not because it's sexy or a vibe, but because it's a guardrail that keeps you from telling yourself bedtime stories with charts.

Actionable metrics tie back to an assumption. Vanity metrics are what you show your friends.

Top-line numbers lie the same way whether a person or a model dressed them up. I once dug into churn at a software company with a real "our support team sucks" culture — support got blamed for everything, and the churn numbers backed it up on the surface. So I went digging myself. One sales rep kept crushing his goals, quarter after quarter, and management loved him for it. Turned out his customers reliably churned right after the quarter closed and his bonus got paid. He was selling services the company didn't actually have, then handing the fulfillment problem to support and services, who obviously couldn't deliver something that didn't exist. By the time the customer walked, he'd already cashed the bonus and moved to his next target. The topline metric — deals closed, quota hit — looked fantastic. The actual business was on fire underneath it, and the only reason anyone caught it was because someone bothered to look past the number to what happened three months later.

That's the whole problem with vanity metrics, AI-generated or not. They measure the moment you want measured, and nothing after it.

Innovation accounting is boring on purpose

If you want one habit that keeps you sane, do cohort analysis. Not "weekly active users" as a single blob. Real cohorts you can compare without squinting.

Signup week cohorts. Activation cohorts. Paid conversion cohorts. Expansion cohorts. Whatever maps to your product.

Here's the basic idea in SQL-ish form:

SELECT
  signup_week,
  COUNT(*) AS signups,
  AVG(activated) AS activation_rate,
  AVG(retained_week4) AS week4_retention,
  AVG(paid) AS paid_conversion,
  SUM(revenue_30d) / COUNT(*) AS rev_per_signup_30d
FROM user_cohorts
GROUP BY 1
ORDER BY 1;

AI should not be doing that math. Regular software should.

Let your warehouse, notebooks, or BI tool calculate the numbers. Add tests. Reconcile against Stripe. Compare sources. Catch the quiet bugs that make dashboards lie politely.

Then use AI to help you read the output like a human who has to make decisions.

Where AI actually helps (and where it doesn't)

AI is good at synthesis. It's bad at being the source of truth.

Use AI for synthesis and consistency

Useful. It saves time. It keeps the team looking at the same facts instead of inventing their own versions.

Use normal software for the money and the math

If you let AI "calculate" retention from a screenshot of a dashboard, you're volunteering to have a bad week.

The human part you can't outsource

Most teams avoid the moment where you admit the thing didn't work.

Not because people are dumb. Because people get attached. You spent a week on it. You argued about it. You shipped it. You want it to be true.

I worked for a startup with a CEO who had a full-on cult of personality thing going. People would nod along to ideas they knew were wrong because the social cost of disagreeing was higher than the cost of being incorrect. I'm not built for that. I ask too many questions and I'm bad at pretending not to notice.

Metrics discipline is the same muscle. It's the willingness to ask "come to our office and watch us work" out loud instead of nodding and moving on. It's the willingness to check what the star sales rep's customers look like in month four instead of admiring his quota in month one.

If activation didn't move, the feature didn't work. If retention didn't move, your "improved onboarding" was probably just new copy. If revenue didn't move, your "growth loop" was a loop in a slide deck.

AI can write a tidy summary of the results. You still have to look at the numbers and say, "Yeah, we're done here."

A simple operating cadence that works

If you want something practical:

  1. Pick one metric that maps to one assumption (activation, retention, paid conversion, expansion)
  2. Track it by cohort (not blended)
  3. Write the experiment hypothesis before you ship (see building MVPs that generate customer evidence for more on validating assumptions early)
  4. Review weekly with a short memo
  5. Kill things fast when the metric doesn't move

If you want help setting this up-cohort dashboards, event design, experiment logging, and the boring parts that keep the numbers honest-I do consulting around analytics, growth loops, and operating cadence. If you show me your dashboard, I'll tell you where it's lying, and I'll be specific about how it's doing it.

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from Startups

View all →
Startups

What Actually Goes in a Consulting Contract

Sep 17, 2026
Startups

Productizing a Service: From Hours to Licenses

Jul 26, 2026
Startups

The MVP That Is Not a Product

Jul 11, 2026