I've watched startups die for all kinds of reasons: no traction, lost investment, and sometimes outright fraud.
Deployments didn't kill any of them.
Deployment dysfunction, though? That's a runway-eater. It burns the one thing you never get back: time. Time you could spend talking to customers. Time you could spend fixing the product. Time you could spend figuring out why nobody is buying the thing you built.
I've watched teams spend weeks building perfect deployment infrastructure while their product sat there untouched. I've seen engineers argue about Kubernetes settings instead of shipping features. I've been in meetings about CI/CD tools when we should have been chatting with customers.
These sins rarely kill you directly. They slow you down enough that something else kills you first.
Sin #1: Staging/Production Drift (The Ghost Bug Sin)
If staging doesn't match production, you're testing a different product. You'll still learn something. You just won't like the lesson.
What it looks like in real life
At one company, I watched a team spend three days debugging a case-sensitivity issue. It only happened in production.
Dev was using utf8_general_ci locally. Production was using utf8_bin.
Case-insensitive collation is forgiving. Binary collation is not here to be your friend. It's here to store data correctly and judge you silently.
The bug looked like this:
- A user signs up fine in dev.
- In prod, the same user gets a weird "already exists" error.
- Everyone starts blaming the auth provider, the network, and then we get philosophical.
Here's the self-aware moment: I initially blamed caching. I always blame caching. It wasn't caching. It was the database collation difference, sitting there like a tiny configuration landmine.
The details change, but the movie is always the same:
1) "It worked on my computer."
2) Hours of debugging.
3) The root cause is that your environments are different.
Why teams let this happen
Matching environments feels "optional" early on. People say:
- "Staging is close enough."
- "We'll fix it later." (Later is busy.)
- "It's a small difference." (It isn't.)
Startups love speed. Speed without consistency is chaos with better branding.
Fixes that actually work
- Use the same database engine everywhere (dev/staging/prod).
- Use the same major versions of runtimes (Python, Node, etc.).
- Use containers if you can, but don't containerize your way into a new religion.
- Make staging "production, but smaller." Same shape, less scale.
Rule of thumb: If staging isn't production-except-scale, expect ghost bugs.
Sin #2: Terrible Secrets Management (The "Oops We Got Hacked" Sin)
If a secret lands in git history, assume it's compromised. Git is forever. Bots don't sleep. Attackers don't need motivation. They need opportunity.
The fastest way to learn this lesson
At one SaaS company, a dev committed an .env file "just for a minute" to help another dev get set up. It included an AWS access key.
Within hours, the AWS bill started climbing. Fast.
I've seen AWS bills hit five figures overnight. No fancy hack. No movie plot. A leaked key and automated scanning finding it quickly.
And yes, you can rotate keys. But you can't rotate time. You'll spend days untangling:
- what got accessed
- what got exfiltrated
- what permissions that key had
- what else that key could see
- what legal/compliance wants from you now
Why it keeps happening anyway
Secrets feel like "plumbing." Nobody gets promoted for plumbing. People are under pressure, onboarding is messy, and everyone wants the app to run right now.
So someone does the shortcut:
- paste key into Slack
- commit it "temporarily"
- store it in a Notion doc
- hardcode it "just for now"
("Just for now" is the official slogan of future incidents.)
In a recent GitGuardian report, they describe secret exposure at the scale of tens of millions of secrets found in public places like GitHub. That's not a rounding error. That's an ecosystem.
A sane baseline
- Add
.envto.gitignoreon day one. - Use a real secrets manager (AWS Secrets Manager or Doppler-pick one).
- Use short-lived credentials where possible.
- Rotate credentials regularly (and immediately after any leak).
- Add secret scanning to CI so you catch it before it merges.
Rule of thumb: If a secret touches git history, treat it as burned. Rotate it.
Sin #3: Skipping Real Staging Tests (The "It Worked On My Machine" Sin)
If you don't click through staging like a user, you are not testing. You are hoping.
My Internet Explorer era (sad but educational)
I worked at a web design firm where meetings multiplied when nobody was shipping.
Firefox had just been released. I was an early adopter, because of course I was. New shiny thing. Faster. Better dev tools. I was living my best life.
We had an e-commerce site launching. It was a big weekend. And on Friday, there was a "patch it real fast" fix.
I tested in Firefox. It looked great. I deployed.
Then the support tickets started flooding in. Checkout was broken for a huge chunk of users. The site was effectively down, but only for the people who mattered: the ones trying to buy things.
The issue was Internet Explorer. Which, at the time, was most of the browser market. I had tested in the browser I liked, not the browser users had.
That was my first major bonehead bug. It taught me:
- test like a user
- don't do "quick fixes" on Friday
- your preferences are not the market
Why smart people still skip this
Staging tests feel slow. People are tired. Deadlines are real. The fix looks small. Everyone wants to ship.
Developers are optimistic by nature. My accountant already hates me for other reasons, so I try not to add "avoidable outage" to the list.
The boring routine that saves you
Have a boring staging routine. Boring is good. Boring means you sleep.
A simple checklist:
- Deploy to staging
- Run the money path end-to-end (for example: signup - confirm email - login - upgrade plan)
- Check logs for new errors
- Verify migrations ran
- Confirm you can redeploy the last known good version
If you can automate this, even better. But even if you can't, do one real user flow with your own hands.
Rule of thumb: If you didn't test it like a user in staging, you didn't test it.
Sin #4: Backups You Never Restore (The "Aspirational Backup" Sin)
Backups you haven't restored from are not backups. They're comforting stories you tell yourself.
The backup that was decor
I worked with a company that did "everything right" on paper. They had backups. They had a schedule.
They tested restores annually.
Then they had a real incident and needed to restore. It turned out months of backups were silently corrupted. They'd been backing up garbage faithfully, every day, like a reliable robot doing the wrong job.
So now they had:
- a production outage
- angry customers
- leadership asking for ETAs
- and backups that were decorative, like a fire extinguisher full of confetti
Restoring wasn't a "click button, problem solved" moment. It became a forensic project.
Why this gets deprioritized
Backups are invisible when they work. They don't ship features. They don't impress investors. They don't show up in demos.
Restore testing feels like a chore... until it becomes the only thing that matters.
Minimum viable safety
- Automated daily backups
- Off-system storage (different account/bucket/region if possible)
- Regular restore tests (monthly is a good start)
- Track two numbers: how much data you can lose, and how long restore takes
If you have a database, you need to practice restoring it. Not once a year. Regularly.
Rule of thumb: A backup is only real after you've restored it successfully.
Sin #5: No Rollback Plan (The "Fix Forward Forever" Sin)
If you can't rollback quickly, your deployment process is broken.
The Thursday night deploy that ate Friday
At a startup that was moving fast (they always are), we did a Thursday night deploy. It included code changes and a database migration.
Friday morning, customers started screaming. The new version was breaking a critical workflow. Support was drowning. Slack was a crime scene.
So we said the magic words: "Let's rollback."
Except... the database schema had changed. Rolling back the code meant the old code would now be talking to a database shape it didn't understand.
So we were stuck with "fix forward." During an outage. While customers are screaming. Fun.
One more detail that still hurts: earlier in my career, I watched someone run a "small" schema change on what they thought was staging. It was the billing database. On the wrong host. Billing went down. The company learned, in real time, what cashflow dependence feels like.
Why rollback gets ignored
Teams treat rollback as a "nice-to-have." They assume the deploy will work. Or they assume they can patch quickly.
Schema changes feel harmless. Someone adds a column. Someone renames a field. Someone drops a table "we don't use anymore." (You do. You just don't know it yet.)
Sometimes the migration is the point of no return.
How to make rollback real (not theoretical)
- Deploy in small steps.
- Prefer backwards-compatible migrations:
- add new columns first
- write code that supports both old and new
- backfill data
- only then remove old columns later
- Version your deploys so you can redeploy the last known good build fast.
A rollback should look like this (plain English: "deploy the previous known-good version"):
git tag prod-2026-01-25
git push --tags
# rollback = redeploy the previous tag
And migrations should support downgrade when possible (plain English: "you can undo the schema change if you must"):
def upgrade():
op.add_column("users", sa.Column("new_field", sa.String(255)))
def downgrade():
op.drop_column("users", "new_field")
One quick aside: tags and migrations are never as clean in real life as they are in blog posts. That's why you practice the rollback before you need it.
Downgrades aren't always safe (data loss is real), but the act of writing them forces you to think clearly about what you're doing.
Rule of thumb: If you can't rollback in single-digit minutes, you have a gamble.
Bad deployments won't usually kill your startup. They'll waste enough time that you miss your window, burn your runway, and end up in "death flailing" mode.
Remember the ants: tiny problems that multiply fast once they get into production. If you ignore them, you don't get "a few ants." You get fire ants.
Sethers
