---
title: "Vanity Metrics Look Better When AI Generates Them"
description: "AI can generate endless signups, landing pages, and charts—but vanity metrics aren't progress. Learn why cohort analysis beats dashboard theater, where AI actually helps (synthesis vs. math), and how to build metrics discipline that catches when features don't work before you waste another week pretending they do."
image: "https://cdn.sethserver.com/blog-images/vanity-metrics-look-better-when-ai-generates-them-social.webp"
---

# Vanity Metrics Look Better When AI Generates Them

AI can generate a lot of stuff.

Landing pages. Cold emails. "Personalized" outreach that reads like a fortune cookie. Product tours. Help docs. Even little charts that look like you're doing science.

And yes, it can generate signups too. Aim it at the top of the funnel, tell it to spray output everywhere, and it will.

I ran into the human version of this problem years before AI could do any of it, at a SaaS company that claimed to use AI to automate mundane, error-prone financial work. A few weeks in, I started asking basic questions. Where's the model? Where's the training data? Where are the evals? The support and services team, based in South Asia, didn't send documentation. They told me to come to their office and watch them work.

Not a joke. That was the actual answer to "where's the model."

So I did a code review instead of a plane ticket. The "AI" was mapping spreadsheet columns and reformatting data. That's it. Excel cosplay wearing a machine learning costume to the investor meeting. And underneath that, there was a second fight nobody put in the pitch deck: an internal "lowest cost" battle, our own support team against a batch of external contractors, both quietly serving the same customers on the side. Everybody racing to be the cheapest labor attached to a product that was mostly labor already. The dashboard still looked great through all of it. The story still sold. The numbers still moved. Just not for the reasons anyone in the board meeting wanted to admit out loud.

Here's the part that actually stuck with me: humans beat computers when humans are cheaper. Every time somebody tells you a workflow is "AI-powered," ask what it costs to run versus what it costs to pay a person in a lower-wage market to do the same task by hand. If the human is cheaper, the human is doing it, and the model is a slide.

AI makes the newer version of that pattern easier, and cheaper to fake at scale. Output gets cheap, so it's easy to confuse activity with progress.

## The chart got prettier. The business didn't.

Vanity metrics love volume:

- More signups
- More demos booked
- More pages published
- More emails sent
- More "experiments" run

AI can pump all of those. You can ship three new onboarding flows in a week and feel like a machine.

But if your activation rate stays flat, you didn't improve onboarding. You improved the number of onboarding flows.

I still like Lean Startup's "innovation accounting" framing. Not because it's sexy or a vibe, but because it's a guardrail that keeps you from telling yourself bedtime stories with charts.

Actionable metrics tie back to an assumption. Vanity metrics are what you show your friends.

Top-line numbers lie the same way whether a person or a model dressed them up. I once dug into churn at a software company with a real "our support team sucks" culture — support got blamed for everything, and the churn numbers backed it up on the surface. So I went digging myself. One sales rep kept crushing his goals, quarter after quarter, and management loved him for it. Turned out his customers reliably churned right after the quarter closed and his bonus got paid. He was selling services the company didn't actually have, then handing the fulfillment problem to support and services, who obviously couldn't deliver something that didn't exist. By the time the customer walked, he'd already cashed the bonus and moved to his next target. The topline metric — deals closed, quota hit — looked fantastic. The actual business was on fire underneath it, and the only reason anyone caught it was because someone bothered to look past the number to what happened three months later.

That's the whole problem with vanity metrics, AI-generated or not. They measure the moment you want measured, and nothing after it.

## Innovation accounting is boring on purpose

If you want one habit that keeps you sane, do cohort analysis. Not "weekly active users" as a single blob. Real cohorts you can compare without squinting.

Signup week cohorts. Activation cohorts. Paid conversion cohorts. Expansion cohorts. Whatever maps to your product.

Here's the basic idea in SQL-ish form:

```sql
SELECT
  signup_week,
  COUNT(*) AS signups,
  AVG(activated) AS activation_rate,
  AVG(retained_week4) AS week4_retention,
  AVG(paid) AS paid_conversion,
  SUM(revenue_30d) / COUNT(*) AS rev_per_signup_30d
FROM user_cohorts
GROUP BY 1
ORDER BY 1;
```

AI should not be doing that math. Regular software should.

Let your warehouse, notebooks, or BI tool calculate the numbers. Add tests. Reconcile against Stripe. Compare sources. Catch the quiet bugs that make dashboards lie politely.

Then use AI to help you read the output like a human who has to make decisions.

## Where AI actually helps (and where it doesn't)

AI is good at synthesis. It's bad at being the source of truth.

### Use AI for synthesis and consistency

- **Churn synthesis**
  Summarize cancellation notes, exit interviews, and support tickets. Pull themes. Quote real sentences. Don't let it invent reasons.
- **Experiment logs**
  Keep the relationship between hypothesis - change - metric - result intact. If you don't write the hypothesis down, you'll "remember" later that the feature was always meant to do whatever happened.
- **Weekly metrics memos**
  A readable one-pager: what moved, what didn't, why we think it happened, and what we're doing next.
- **Cohort dashboard narration**
  "Activation down 3 points for the March 10 cohort. Retention flat. Paid conversion up slightly. Top churn reason: missing integrations."

Useful. It saves time. It keeps the team looking at the same facts instead of inventing their own versions.

### Use normal software for the money and the math

- **Cohort calculations** (SQL, product analytics, notebooks)
- **Revenue reporting** (Stripe + accounting, not vibes)
- **Experiment tracking** (feature flags, exposure logging, event pipelines)
- **Data validation** (tests, reconciliations, source comparisons)

If you let AI "calculate" retention from a screenshot of a dashboard, you're volunteering to have a bad week.

## The human part you can't outsource

Most teams avoid the moment where you admit the thing didn't work.

Not because people are dumb. Because people get attached. You spent a week on it. You argued about it. You shipped it. You want it to be true.

I worked for a startup with a CEO who had a full-on cult of personality thing going. People would nod along to ideas they knew were wrong because the social cost of disagreeing was higher than the cost of being incorrect. I'm not built for that. I ask too many questions and I'm bad at pretending not to notice.

Metrics discipline is the same muscle. It's the willingness to ask "come to our office and watch us work" out loud instead of nodding and moving on. It's the willingness to check what the star sales rep's customers look like in month four instead of admiring his quota in month one.

If activation didn't move, the feature didn't work. If retention didn't move, your "improved onboarding" was probably just new copy. If revenue didn't move, your "growth loop" was a loop in a slide deck.

AI can write a tidy summary of the results. You still have to look at the numbers and say, "Yeah, we're done here."

## A simple operating cadence that works

If you want something practical:

1. **Pick one metric that maps to one assumption** (activation, retention, paid conversion, expansion)
2. **Track it by cohort** (not blended)
3. **Write the experiment hypothesis before you ship** (see [building MVPs that generate customer evidence](https://www.sethserver.com/startups/the-mvp-that-is-not-a-product.html) for more on validating assumptions early)
4. **Review weekly** with a short memo
5. **Kill things fast** when the metric doesn't move

If you want help setting this up-cohort dashboards, event design, experiment logging, and the boring parts that keep the numbers honest-I do consulting around analytics, growth loops, and operating cadence. If you show me your dashboard, I'll tell you where it's lying, and I'll be specific about how it's doing it.

-Sethers


```json
{"@context": "https://schema.org", "@type": "BlogPosting", "headline": "Vanity Metrics Look Better When AI Generates Them", "datePublished": "2026-07-11T02:17:41", "dateModified": "2026-09-17T19:48:54", "image": {"@type": "ImageObject", "url": "https://cdn.sethserver.com/blog-images/vanity-metrics-look-better-when-ai-generates-them-social.webp", "width": 400, "height": 400}, "url": "https://www.sethserver.com/startups/vanity-metrics-look-better-when-ai-generates-them.html", "author": {"@type": "Person", "name": "Seth Black", "url": "https://www.linkedin.com/in/seth-black-tx"}, "publisher": {"@type": "Organization", "name": "Sethserver", "logo": {"@type": "ImageObject", "url": "https://www.sethserver.com/static/images/PaperAirplane512.png"}}, "description": "AI can generate endless signups, landing pages, and charts—but vanity metrics aren't progress. Learn why cohort analysis beats dashboard theater, where AI actually helps (synthesis vs. math), and how to build metrics discipline that catches when features don't work before you waste another week pretending they do.", "articleSection": "Business"}
{"@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "sethserver", "item": "https://www.sethserver.com/"}, {"@type": "ListItem", "position": 2, "name": "Startups", "item": "https://www.sethserver.com/startups/"}, {"@type": "ListItem", "position": 3, "name": "Vanity Metrics Look Better When AI Generates Them", "item": "https://www.sethserver.com/startups/vanity-metrics-look-better-when-ai-generates-them.html"}]}
```
