---
title: "The Power of Simplicity: How to Use S3 for Better Unstructured Data Storage"
description: "Learn why S3 is often the best choice for storing unstructured data like logs, sensor payloads, and JSON blobs. Discover common pitfalls with relational and NoSQL databases, practical S3 key structure patterns, and how to avoid key drift, permission issues, and lifecycle management mistakes when building scalable data storage systems."
image: "https://cdn.sethserver.com/blog-images/the-power-of-simplicity-how-to-use-s3-for-better-unstructured-data-storage-social-1767152298.webp"
---

# The Power of Simplicity: How to Use S3 for Better Unstructured Data Storage

One of the dumbest architectural decisions I've ever made was at Atlas Wearables when I replaced a perfectly good S3-backed CSV store with a relational database.

Why was it dumb? Relational databases are great at structured data. Wearable sensor dumps are not that. Yes, we have JSON columns now. But that's a quick hack for *some* unstructured settings, not a plan for storing and retrieving gigabytes of time-series-ish blobs that keep changing shape.

## S3 is probably good enough

Unstructured data piles up fast. Logs. Sensor payloads. Scraped text. Random JSON from partners. "Temporary" exports that become a permanent part of your identity.

When that happens, teams usually reach for:

- Postgres/MySQL because it's familiar
- MongoDB/DynamoDB because "it stores JSON"
- Or the thing that feels almost suspiciously simple: S3 + a clean key structure

I'm going to make the case for S3. Because it's simple and hard to screw up.

Also because I've watched [startups](https://www.sethserver.com/startups/what-is-a-startup.html) light weeks on fire trying to design the Perfect Data Model for data nobody has even looked at yet. The database didn't fail. The calendar did.

## Here's what you're choosing between

### Relational databases (Postgres/MySQL)

If you need joins, constraints, and real querying across well-defined fields, relational databases are excellent. If you're storing large blobs that change format every few weeks, you'll end up with a "JSON column" that becomes a junk drawer. You *can* index it. You *can* migrate it. You *can* also spend your whole Friday doing it, then watch the site burn down. Ask me how I know.

### NoSQL (MongoDB/DynamoDB)

Document stores can feel like freedom at first: "Just shove the JSON in there." Then you learn about access patterns, hot partitions, query limits, and cost surprises. And even if you do this well, you often still end up putting the big payloads in... S3. Because big payloads are what S3 is for. MongoDB sales pitches are persuasive, but they don't pay your AWS bill. I'm going to be honest, I'm still not sold on Mongo. It feels like one of those tools - like the tool that pulls the rear bearings off a 2012 Nissan Frontier. It is literally a lifesaver...if you own a 2012 Nissan Frontier...and you're needing to pull the bearings off the rear axle.

### S3 (object storage)

S3 is cheap, durable, and boring in the best way. It's not a database. You don't "query" it in the same way. You store objects, you fetch objects. If you need analytics later, you add Athena or an index somewhere else. Most teams don't need fancy queries on raw unstructured data sitting at rest. They need to store it safely, retrieve it predictably, and process it.

## A concrete example: unstructured weather snapshots

Weather data isn't the best example because it *can* be structured. But it's easy to picture, and the "shape" changes often (new sensors, new fields, weird vendor formats). So we'll use it.

Let's say each user can upload weather snapshots from different sources. Store it like this:

- `s3://my-bucket/<station_guid>/metadata.json`
- `s3://my-bucket/<station_guid>/snapshots/<snapshot_guid>.json`
- `s3://my-bucket/<station_guid>/raw/<snapshot_guid>.json.gz` (optional)

Example `metadata.json`:

```json
{
  "user_guid": "u_123",
  "created_at": "2025-12-01T12:00:00Z",
  "snapshots": [
    {"snapshot_guid": "w_001", "ts": "2025-12-01T11:59:00Z", "source": "station_A"}
  ]
}
```

Nothing fancy here. Pretty straightforward, right? That's the point.

## Why this works (in real life)

The biggest win is you stop pretending you know the final schema on day one. You store the raw thing as-is, with a key layout you can reason about. When formats evolve, you don't do migrations. You just write the next snapshot with the next shape.

Reads stay predictable too. You fetch a small metadata object, then fetch the specific snapshot objects you need. No "scan the table and pray." No "we need three compound indexes and a sacrifice."

And when you *actually* need indexing, you add it where it belongs: derived fields in Postgres, DynamoDB, OpenSearch, whatever. The raw data stays in S3 as your source of truth. I've done this pattern in multiple messy startups, including ones where teams tried to "pivot" mid-flight. The ones that survived usually did less, not more.

## What actually goes wrong in practice (and how to avoid it)

- **Key structure drift**: people start dumping files wherever. Pick a convention early (`/<user_guid>/...` or `/<org_guid>/...` or `/<station_guid>/...` or even nest them) and treat it like an API.
- **No lifecycle rules**: raw data grows forever. Add lifecycle policies (archive or delete) based on real retention needs. Most companies don't need to keep everything forever. They just *feel* like they do.
- **Permissions get sloppy**: use IAM policies so a user or service can only access its own prefix (`/<user_guid>/...`). This is where "simple" can still bite you if you're careless.
- **No versioning / no recovery plan**: turn on versioning if the data matters. It's cheap insurance compared to explaining to a customer why last week's data is "gone but, like, gone gone."

If your access pattern is basically "store it, fetch it, process it," [S3 is usually the right first answer](https://www.sethserver.com/programming/connecting-your-replit-app-to-aws-s3.html). You can always add a database later. It's much harder to subtract one after the team has built a small religion around it.

No go forth and rack up an insane S3 bill!

-Sethers


```json
{"@context": "https://schema.org", "@type": "BlogPosting", "headline": "The Power of Simplicity: How to Use S3 for Better Unstructured Data Storage", "datePublished": "2025-12-31T06:10:35", "dateModified": "2026-09-17T19:47:55", "image": {"@type": "ImageObject", "url": "https://cdn.sethserver.com/blog-images/the-power-of-simplicity-how-to-use-s3-for-better-unstructured-data-storage-social-1767152298.webp", "width": 400, "height": 400}, "url": "https://www.sethserver.com/programming/the-power-of-simplicity-how-to-use-s3-for-better-unstructured-data-storage.html", "author": {"@type": "Person", "name": "Seth Black", "url": "https://www.linkedin.com/in/seth-black-tx"}, "publisher": {"@type": "Organization", "name": "Sethserver", "logo": {"@type": "ImageObject", "url": "https://www.sethserver.com/static/images/PaperAirplane512.png"}}, "description": "Learn why S3 is often the best choice for storing unstructured data like logs, sensor payloads, and JSON blobs. Discover common pitfalls with relational and NoSQL databases, practical S3 key structure patterns, and how to avoid key drift, permission issues, and lifecycle management mistakes when building scalable data storage systems.\r\n", "articleSection": "Technology"}
{"@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "sethserver", "item": "https://www.sethserver.com/"}, {"@type": "ListItem", "position": 2, "name": "Programming", "item": "https://www.sethserver.com/programming/"}, {"@type": "ListItem", "position": 3, "name": "The Power of Simplicity: How to Use S3 for Better Unstructured Data Storage", "item": "https://www.sethserver.com/programming/the-power-of-simplicity-how-to-use-s3-for-better-unstructured-data-storage.html"}]}
```
