I stopped trusting coding quizzes the day I hired someone who aced ours and then couldn't write a single function on the job. Not one.
They passed the technical screen. They passed the personality interview. They said all the right things. Then I gave them actual work. Nothing came back. I tried pairing with them and they froze completely. Not "nervous." Not "needs a minute." Just... stuck.
After a little digging, the story got simple: they couldn't program at all. They'd somehow gamed the quiz, got themselves into a "cool startup," and figured they'd learn on the job before anyone caught on. Spoiler: they didn't.
That was the last time I used a quiz as my main filter.
The two kinds of "code quizzes" that waste everyone's time
1) Brainteasers
You know the ones. "How many golf balls fit in a 747?" These questions mostly measure whether someone has practiced that exact kind of question. Google famously said they're a waste of time. I believe them. I've also watched interviewers ask them because it lets the interviewer feel clever for five minutes.
If you want to know if someone can ship Python, asking them to estimate the mass of the sun tells you absolutely nothing.
2) LeetCode-style algorithm screens
LeetCode screens aren't completely worthless. They're just terrible at predicting who can actually do the job.
They're good at finding people who memorized 200 LeetCode problems. That's it.
They also select for "interview athletes": people who spent three months grinding algorithm problems that have zero overlap with your actual codebase. Meanwhile, solid engineers who get anxious while someone watches them type can look worse than they are.
And you can run a "fair" algorithm screen and still miss the only question that matters: can they do the actual job you're hiring for?
Most startup work is not "reverse a binary tree in 12 minutes." It's "read an ugly codebase, find the real bug, fix it without breaking five other things, and leave it cleaner than you found it."
What I do instead: a conversational interview with hard edges
I like structure. I just don't like theater.
The best setup I've found is simple. Two parts:
- A competency scorecard
- A code review
That's it.
Part 1: Competency scorecard (but keep it human)
I keep a short list of topics I actually care about: Python basics, debugging, logging, exceptions, and AWS if the job needs it.
Then I ask questions that give people room to show what they know.
Example:
"Half the repo uses
print(), the other half useslogging. How would you fix that without breaking everything?"
A solid Python dev will talk about log levels, handlers, formatting, migration plans, and incremental cleanup.
Someone who's faking it will say "yeah I'd standardize it" and pray you change topics.
I'm not trying to interrogate anyone. I just want enough conversation that real experience becomes obvious.
And yes, I fill out the scorecard right then. If I wait, my brain will invent a completely different interview. It always does.
Part 2: The code review (my favorite lie detector)
Code reviews work better than almost anything else I've tried.
They're collaborative. They're bounded. They look like real work.
I keep a small rotation of short Python scripts that do normal things:
- Parse JSON
- Touch S3 objects
- Simple client/server sockets
- A small script that moves data around
Then I ask the candidate to review it with me.
What I'm actually looking for:
- Do they notice error handling holes?
- Do they mention testing without giving me a 20-minute lecture?
- Can they explain tradeoffs clearly?
- Can they prioritize, or do they get lost polishing the wrong thing?
This catches something specific: people who can talk but can't ship.
I worked at a SaaS startup once that sold "AI-powered" financial automation. After a few weeks I asked to see the AI. They kept dodging. Finally they flew me to an overseas office to "see the team work." The AI was people copying data between spreadsheets. That's it.
Code review has a nice property: if the product is real, it holds up. If it's smoke and mirrors, that shows up too.
The checklist: how to stop burning time
Here's what I try to hold myself to:
- No puzzles. They don't predict the job.
- Keep it short. Nobody needs 17 rounds of interviews.
- Ask everyone the same questions. Your gut is lying to you.
- Use work samples. Code review is a work sample that fits in 30โ45 minutes.
- Treat people like humans. I watched a company do a team retreat on Friday and lay off half the team on Monday. Don't do that.
Hiring is messy. People lie, people freeze up, people game the system. But you can still build something that works. Stop asking trivia questions. Start looking at whether they can actually do the job.
-Sethers