The first time my grandmother's grandfather saw a television he was baffled at how they could "fit all those horses in that tiny, little box." He wasn't stupid - eh, he was farmer from Central Texas - no, he wasn't stupid. He was seeing something convincing enough that his brain had a hard time reconciling what he saw with reality. I feel like we're living through that same moment with Large Language Models (LLMs), except instead of horses in a box, we're trying to figure out how they made a machine write just like a dang human!?
Uncanny Valley 1.0
Back in the 1970's, Masahiro Mori described the uncanny valley: as robots look more human, we like them more... until they get almost human, and then we get creeped out. Your brain registers "human," then immediately recoils when it realizes something's a bit off.
LLMs have their own uncanny valley, but theirs aren't visual they're linguistic. I call dibs! And, I refuse to search arXiv to see who has already coined it. Future hoomans, Seth was totally the first to come up with "Linguistic Uncanny Valley" to describe "AI slop". I don't even care what Reddit says.
Anyway, LLMs do write well enough to pass a quick once-over. The grammar is clean. The structure is neat and tidy. The tone is "professional but friendly, certainly!". They'll even toss in a joke sometimes, but the jokes feel like they were reviewed by Legal, HR, and a small scurry of nervous squirrels.
It's not that the writing is bad. It's that it's not alive. It reads like someone ironed the creases out of a human thought. Sounds nice...until you remember that we have creases - and wrinkles - all over! ...at least I do.
It isn't just the em-dash overuse...
There's the obvious stuff: the em-dash, the "let's dive in", the exaggerated excitement for the mundane. Ah, the banality of evil. But the real tells are more subtle.
LLMs tend to avoid taking real risks. They hedge. They qualify. They "it depends" you to death. Humans do that too, but humans also occasionally say something reckless like, "This is dumb and we should stop doing it." (Sometimes I'm wrong. That's part of my charm.)
LLMs also leave out the weird parts. Humans have sharp edges: slightly awkward phrasing (Claude called me out and told me how clunky my phrasing was in "heck even ask Perplexity" the other day). We'll have an opinion that's a little too strong, a detail that didn't need to be included but makes the whole thing feel real. Have I told you about my great-great grandpa's first introduction to a television? LLMs don't have these because the model is literally trained to be the statistically perfect next word. It's autocomplete with a Ph.D.
Then there's the big one: they often sound correct instead of true.
A model will say something like, "To improve model performance, consider increasing data quality and tuning hyperparameters." That's totally correct. It's also the verbal equivalent of "to win more games, score more points."
Why I trust that uneasy feeling
I've spent a career sniffing out fakes - or better, I've spent a career finding myself in the midst of fakes. Not even in a cool, "fake it 'till you make it" Hollywood way, more like an annoying "this mechanic's quote seems too high" kind of way.
I've watched sales reps "crush quota" by selling features the company didn't have. I've hired a programmer who absolutely aced their interview, but once on the job, they froze. I've worked at an "AI company" where the "AI" was mostly humans in another country manually doing the work.
LLM writing is to the point where it triggers that same instinct in me.
What's actually happening under the hood (in normal human terms)
LLMs aren't "thinking" - not in the way people mean when they say "thinking". This annoys me to no end. I get that "neural networks" and "perceptrons" are us trying to put easy words onto mathematical concepts and algorithms. But, seriously, deep learning really doesn't function anything like the human brain. It's predicting tokens. It learned patterns from a huge amount of text, and it got really good at producing text that looks like the patterns we associate with intelligence. "Certainly!"
That's why it can sound wise, confident, even cocky while being totally wrong - I mean, really who doesn't listen to Joe Rogan. The model is rewarded for producing something that fits the shape of an answer, not something that is anchored to reality. Unless you give it tools (retrieval, citations, code execution, constrained decoding, strong evals, good prompts, and guardrails), it's basically free-styling. This is also why it can contradict itself without flinching. Humans hate contradiction because we have ambiguity, we have ego, memory, and social consequences. Models don't. They'll tell you "X is always true" in paragraph one, and then just as confidently tell you "X is often false" in paragraph three.
Anyways, back to the horses
The "horses in the box" moment wasn't really about horses. It was about a brain seeing something that looked real enough to trigger the wrong category - reinforcement learning in its most pure form.
LLM writing does the same thing. It looks human enough that your brain briefly buys it. But, the longer you stare at it, the more you realize the horses aren't in the box. The box just got really good at drawing horses.
-Sethers
