sethserver.com Subscribe
Robot swimming through a pool, but instead of water it has documents

The Power of Simplicity: How to Use S3 for Better Unstructured Data Storage

By Seth Black • Updated: September 17, 2026

Programming · 4 min read

Like this kind of writing? Get one email a week with notes on startups, AI, and the occasional strong opinion about Python: subscribe to the newsletter.

One of the dumbest architectural decisions I've ever made was at Atlas Wearables when I replaced a perfectly good S3-backed CSV store with a relational database.

Why was it dumb? Relational databases are great at structured data. Wearable sensor dumps are not that. Yes, we have JSON columns now. But that's a quick hack for some unstructured settings, not a plan for storing and retrieving gigabytes of time-series-ish blobs that keep changing shape.

S3 is probably good enough

Unstructured data piles up fast. Logs. Sensor payloads. Scraped text. Random JSON from partners. "Temporary" exports that become a permanent part of your identity.

When that happens, teams usually reach for:

I'm going to make the case for S3. Because it's simple and hard to screw up.

Also because I've watched startups light weeks on fire trying to design the Perfect Data Model for data nobody has even looked at yet. The database didn't fail. The calendar did.

Here's what you're choosing between

Relational databases (Postgres/MySQL)

If you need joins, constraints, and real querying across well-defined fields, relational databases are excellent. If you're storing large blobs that change format every few weeks, you'll end up with a "JSON column" that becomes a junk drawer. You can index it. You can migrate it. You can also spend your whole Friday doing it, then watch the site burn down. Ask me how I know.

NoSQL (MongoDB/DynamoDB)

Document stores can feel like freedom at first: "Just shove the JSON in there." Then you learn about access patterns, hot partitions, query limits, and cost surprises. And even if you do this well, you often still end up putting the big payloads in... S3. Because big payloads are what S3 is for. MongoDB sales pitches are persuasive, but they don't pay your AWS bill. I'm going to be honest, I'm still not sold on Mongo. It feels like one of those tools - like the tool that pulls the rear bearings off a 2012 Nissan Frontier. It is literally a lifesaver...if you own a 2012 Nissan Frontier...and you're needing to pull the bearings off the rear axle.

S3 (object storage)

S3 is cheap, durable, and boring in the best way. It's not a database. You don't "query" it in the same way. You store objects, you fetch objects. If you need analytics later, you add Athena or an index somewhere else. Most teams don't need fancy queries on raw unstructured data sitting at rest. They need to store it safely, retrieve it predictably, and process it.

A concrete example: unstructured weather snapshots

Weather data isn't the best example because it can be structured. But it's easy to picture, and the "shape" changes often (new sensors, new fields, weird vendor formats). So we'll use it.

Let's say each user can upload weather snapshots from different sources. Store it like this:

Example metadata.json:

{
  "user_guid": "u_123",
  "created_at": "2025-12-01T12:00:00Z",
  "snapshots": [
    {"snapshot_guid": "w_001", "ts": "2025-12-01T11:59:00Z", "source": "station_A"}
  ]
}

Nothing fancy here. Pretty straightforward, right? That's the point.

Why this works (in real life)

The biggest win is you stop pretending you know the final schema on day one. You store the raw thing as-is, with a key layout you can reason about. When formats evolve, you don't do migrations. You just write the next snapshot with the next shape.

Reads stay predictable too. You fetch a small metadata object, then fetch the specific snapshot objects you need. No "scan the table and pray." No "we need three compound indexes and a sacrifice."

And when you actually need indexing, you add it where it belongs: derived fields in Postgres, DynamoDB, OpenSearch, whatever. The raw data stays in S3 as your source of truth. I've done this pattern in multiple messy startups, including ones where teams tried to "pivot" mid-flight. The ones that survived usually did less, not more.

What actually goes wrong in practice (and how to avoid it)

If your access pattern is basically "store it, fetch it, process it," S3 is usually the right first answer. You can always add a database later. It's much harder to subtract one after the team has built a small religion around it.

No go forth and rack up an insane S3 bill!

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from Programming

View all →
Programming

What's the Differences Between Relational, NoSQL, Vectors, and Caches

Apr 14, 2026
Programming

Git and Source Control in AI Land

Apr 14, 2026
Programming

Domains, DNS, and why it feels fragile

Apr 14, 2026