sethserver.com Subscribe
Elderly retro 1970s toy robot with boxy square head, rectangular body, accordion tube arms with claw hands, square head with single antenna on top, two circular eyes, long white beard, holding a very heavy barbell over his head.

Why Old-School ML Models Still Win in Production: Naive Bayes, Logistic Regression & Decision Trees

By Seth Black • Updated: September 17, 2026

AI · 5 min read

Like this kind of writing? Get one email a week with notes on startups, AI, and the occasional strong opinion about Python: subscribe to the newsletter.

The fastest way to lose faith in "modern AI" is to ship it inside a $200 physical gadget and then watch your support inbox light up because your backend is slow.

I worked at a physical product company that rode a successful crowdfunding campaign into the real world. The genius programmer who built the original system used the fame to land a FAANG job and vanished, leaving a crater where the software team used to be. That's where I walked in.

The codebase had all the classics. Files named things like final_final_v3_REAL.py. A flag that looked like it controlled environments but was basically a decorative sticker:

if production:
    production = True

The device connected over Wi‑Fi. If our API took too long, customers didn't think "huh, network latency." They thought the device was broken. And since it was a physical product, "broken" meant refunds, angry emails, and replacement shipping costs. You can't A/B test your way out of FedEx.

That's why old-school ML models still win in production. Not because they're cooler. Because they fit inside reality.

Production doesn't care about your benchmark score

A lot of ML chats end with leaderboard screenshots. Shipping starts with questions like:

On that product team, we didn't have infinite data. We had noisy customer feedback, a small set of labeled cases, and constant pressure to make the thing feel instant. Smaller, specialized models beat bigger general models because the task was narrow and the constraints were brutal.

Naive Bayes: dumb assumption, useful results

Naive Bayes gets made fun of because it assumes features are independent. In text classification, that "naive" part often doesn't hurt much.

If you need to classify support emails into buckets like refund, setup issue, shipping, warranty, Naive Bayes will usually get you surprisingly far with a tiny dataset. It trains fast. It runs cheap. It doesn't demand a GPU altar.

This is the model you use when you have 500 labeled examples and a deadline that doesn't care about your feelings.

Logistic Regression: boring, everywhere, and still a great idea

Logistic regression is the model people stop talking about because it's too normal. Then they ship it anyway.

It takes a weighted sum of features and squashes it through a sigmoid, giving you a probability between 0 and 1. You pick a threshold (often 0.5) and call it a day.

The part that saves you in production is simple: you can read the coefficients. You can tell a non-ML person, "This feature pushes the prediction up, this one pushes it down." That's what "debuggable" looks like when you're under pressure.

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=200)
clf.fit(X_train, y_train)

proba = clf.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)

If the model starts flagging half your good users as fraud, you can inspect what changed. You don't have to perform a séance over attention heads.

Decision Trees and Random Forests: models you can argue with

A decision tree is easy to explain because it's literally a set of questions. You can draw it on a whiteboard and have a productive argument about it.

The downside is trees love to memorize. Give them enough depth and they'll happily learn that "customer_id ends with 7" is a meaningful signal.

Random forests fix a lot of that by training many trees on random slices of data and averaging. They're the "temporary" solution that ends up running for three years because it keeps working and nobody wants to touch it.

Feature importance from a random forest is also a nice reality check. It tells you what signals matter so you can log them properly and stop guessing.

HMMs: old, fast, and weirdly practical

Hidden Markov Models are from the late 1960s. They model sequences by assuming there's a hidden state generating the observations. You observe the outputs, infer the states, and you can inspect transitions.

They're trained with algorithms like Baum-Welch. They used to show up in speech recognition and autocorrect. They still work fine for certain time-series problems where you need something fast and inspectable.

If your system has states like normal, degrading, failing, an HMM can be a clean baseline. And when it fails, it fails in a way you can actually look at.

LLMs taught me the "CEO janitor" lesson

I built an AI survey response system where generated personas answered questions based on persona attributes. The setup was simple: "Your job title is janitor and you make $15,000/yr." Then I asked: "What's your job title?"

The model answered: "CEO."

Asked salary? "$35,000/yr."

Just off-the-wall stuff. Not because it was being sneaky. Because LLMs predict likely tokens based on context. "Janitor" and "CEO" both live near "job" in the token soup. The model wasn't thinking. It was doing probability math and grabbing something that sounded plausible.

That experience made me appreciate older models even more. They're narrow. They fail in simpler ways. You can box them in without writing a 12-page prompt and hoping the model feels cooperative today.

Use the old models when reality is tight

Reach for Naive Bayes, logistic regression, trees, forests, HMMs if:

Transformers and LLMs are useful tools. I like them. I also don't want them anywhere near a system where a slow API makes a physical device look dead.

Pick the model that fits the job. The older stuff still pays rent.

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from AI

View all →
AI

How to be a Good Farrier: What the Car Teaches Us About AI and Your Job

Nov 10, 2026
AI

Principle of Least Privilege in the AI Age

Apr 14, 2026
AI

How to Actually Secure MCP (Model Context Protocol)

Apr 14, 2026