sethserver.com Subscribe
Two friendly yellow robots working together, one pouring liquid from an orange bucket into a smaller container held by the other robot against a mint green and navy blue background - teamwork and collaboration illustration

Replit Object Storage with AWS S3: Python and TypeScript Setup

By Seth Black • Updated: September 17, 2026

Programming · 12 min read

Want more posts like this in your inbox? One email a week, no spam - subscribe to the newsletter.

S3 is one of those AWS services that just works. It's been around since 2006, it's battle-tested, and it scales infinitely without you having to think about disk provisioning or RAID configurations. Need to store files? S3. Need to host static assets? S3. Need cheap storage for data you rarely access? Also S3.

I use S3 constantly for storing JSON and XML documents when training ML algorithms. When you're doing infrequent reads and writes — pull a dataset, train a model, push results back — S3 is perfect. It's dirt cheap for storage, you only pay for what you transfer, and you don't need to keep a database running 24/7 for data that gets touched once a week. For batch ML workflows, it's the obvious choice.

Let's get your Replit app connected.

The Short Version

You need an S3 bucket, IAM credentials with the right permissions, and a few lines of code. Unlike RDS, there's no networking to configure — S3 is a public service (your data isn't public, but the API endpoint is). No security groups, no VPCs, no public accessibility toggles.

Step 1: Create an S3 Bucket

Head to the S3 console in AWS and click "Create bucket."

Click "Create bucket" and you're done with this part.

Step 2: Create IAM Credentials

Your Replit app needs credentials to access S3. We'll create an IAM user with minimal permissions.

  1. Go to IAM in the AWS Console
  2. Click "Users" then "Create user"
  3. Name it something like replit-s3-access
  4. Click "Next"
  5. Choose "Attach policies directly"
  6. For now, search for AmazonS3FullAccess — we'll tighten this up in a moment
  7. Create the user
  8. Click on the user, go to "Security credentials"
  9. Click "Create access key"
  10. Choose "Application running outside AWS"
  11. Save the Access Key ID and Secret Access Key

Tighten Those Permissions

AmazonS3FullAccess gives access to every bucket in your account. That's too broad. Create a custom policy that limits access to just your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Replace your-bucket-name with your actual bucket name. The first ARN allows listing the bucket contents, the second allows operations on objects inside it.

Go back to your IAM user, detach the AmazonS3FullAccess policy, and attach your new custom policy instead.

Step 3: Set Up Secrets in Replit

Add your AWS credentials to Replit's Secrets:

AWS_ACCESS_KEY_ID = your_access_key_id
AWS_SECRET_ACCESS_KEY = your_secret_access_key
AWS_REGION = us-east-1
S3_BUCKET_NAME = your-bucket-name

Step 4: Connect From Your Code

Python with boto3

Basic operations:

import os
import json
import boto3
from botocore.exceptions import ClientError

s3 = boto3.client(
    's3',
    region_name=os.environ.get('AWS_REGION', 'us-east-1'),
    aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
    aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
)

BUCKET = os.environ.get('S3_BUCKET_NAME')


def upload_file(local_path: str, s3_key: str) -> bool:
    """Upload a file to S3."""
    try:
        s3.upload_file(local_path, BUCKET, s3_key)
        print(f"Uploaded {local_path} to s3://{BUCKET}/{s3_key}")
        return True
    except ClientError as e:
        print(f"Upload failed: {e}")
        return False


def download_file(s3_key: str, local_path: str) -> bool:
    """Download a file from S3."""
    try:
        s3.download_file(BUCKET, s3_key, local_path)
        print(f"Downloaded s3://{BUCKET}/{s3_key} to {local_path}")
        return True
    except ClientError as e:
        print(f"Download failed: {e}")
        return False


def list_objects(prefix: str = '') -> list:
    """List objects in the bucket with optional prefix filter."""
    try:
        response = s3.list_objects_v2(Bucket=BUCKET, Prefix=prefix)
        return [obj['Key'] for obj in response.get('Contents', [])]
    except ClientError as e:
        print(f"List failed: {e}")
        return []


def delete_object(s3_key: str) -> bool:
    """Delete an object from S3."""
    try:
        s3.delete_object(Bucket=BUCKET, Key=s3_key)
        print(f"Deleted s3://{BUCKET}/{s3_key}")
        return True
    except ClientError as e:
        print(f"Delete failed: {e}")
        return False

Working with JSON documents (my bread and butter for ML work):

import os
import json
import boto3
from typing import Any
from datetime import datetime

s3 = boto3.client(
    's3',
    region_name=os.environ.get('AWS_REGION', 'us-east-1'),
    aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
    aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
)

BUCKET = os.environ.get('S3_BUCKET_NAME')


def save_json(data: Any, s3_key: str) -> str:
    """Save a Python object as JSON to S3."""
    json_bytes = json.dumps(data, indent=2, default=str).encode('utf-8')
    s3.put_object(Bucket=BUCKET, Key=s3_key, Body=json_bytes, ContentType='application/json')
    return f"s3://{BUCKET}/{s3_key}"


def load_json(s3_key: str) -> Any:
    """Load a JSON document from S3."""
    response = s3.get_object(Bucket=BUCKET, Key=s3_key)
    return json.loads(response['Body'].read().decode('utf-8'))


def save_training_dataset(dataset: list, model_name: str) -> str:
    """Save a training dataset with timestamp for reproducibility."""
    timestamp = datetime.utcnow().strftime('%Y%m%d_%H%M%S')
    s3_key = f"datasets/{model_name}/{timestamp}/training_data.json"
    return save_json(dataset, s3_key)


def save_model_metrics(metrics: dict, model_name: str, version: str) -> str:
    """Save model training metrics."""
    s3_key = f"models/{model_name}/{version}/metrics.json"
    return save_json(metrics, s3_key)


# Example ML workflow
training_data = [
    {"input": [1, 2, 3], "label": "A"},
    {"input": [4, 5, 6], "label": "B"},
    # ... thousands more rows
]

# Save dataset before training
dataset_path = save_training_dataset(training_data, "classifier_v1")
print(f"Dataset saved to {dataset_path}")

# After training, save metrics
metrics = {
    "accuracy": 0.94,
    "f1_score": 0.92,
    "training_time_seconds": 3842,
    "hyperparameters": {"learning_rate": 0.001, "epochs": 100}
}
metrics_path = save_model_metrics(metrics, "classifier_v1", "20240115")
print(f"Metrics saved to {metrics_path}")

Working with XML documents:

import os
import boto3
import xml.etree.ElementTree as ET
from typing import Optional

s3 = boto3.client(
    's3',
    region_name=os.environ.get('AWS_REGION', 'us-east-1'),
    aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
    aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
)

BUCKET = os.environ.get('S3_BUCKET_NAME')


def save_xml(root: ET.Element, s3_key: str) -> str:
    """Save an XML ElementTree to S3."""
    xml_bytes = ET.tostring(root, encoding='utf-8', xml_declaration=True)
    s3.put_object(Bucket=BUCKET, Key=s3_key, Body=xml_bytes, ContentType='application/xml')
    return f"s3://{BUCKET}/{s3_key}"


def load_xml(s3_key: str) -> ET.Element:
    """Load an XML document from S3."""
    response = s3.get_object(Bucket=BUCKET, Key=s3_key)
    return ET.fromstring(response['Body'].read())


# Example: save training configuration as XML
config = ET.Element("training_config")
ET.SubElement(config, "model_type").text = "random_forest"
ET.SubElement(config, "n_estimators").text = "100"
ET.SubElement(config, "max_depth").text = "10"

save_xml(config, "configs/rf_config.xml")

TypeScript with AWS SDK v3

Basic operations:

import { 
  S3Client, 
  PutObjectCommand, 
  GetObjectCommand, 
  DeleteObjectCommand,
  ListObjectsV2Command 
} from '@aws-sdk/client-s3';
import { Readable } from 'stream';
import * as fs from 'fs';

const s3Client = new S3Client({
  region: process.env.AWS_REGION || 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  }
});

const BUCKET = process.env.S3_BUCKET_NAME!;

async function streamToString(stream: Readable): Promise<string> {
  const chunks: Buffer[] = [];
  for await (const chunk of stream) {
    chunks.push(Buffer.from(chunk));
  }
  return Buffer.concat(chunks).toString('utf-8');
}

async function uploadFile(localPath: string, s3Key: string): Promise<void> {
  const fileContent = fs.readFileSync(localPath);
  const command = new PutObjectCommand({
    Bucket: BUCKET,
    Key: s3Key,
    Body: fileContent
  });
  await s3Client.send(command);
  console.log(`Uploaded ${localPath} to s3://${BUCKET}/${s3Key}`);
}

async function downloadFile(s3Key: string, localPath: string): Promise<void> {
  const command = new GetObjectCommand({
    Bucket: BUCKET,
    Key: s3Key
  });
  const response = await s3Client.send(command);
  const content = await streamToString(response.Body as Readable);
  fs.writeFileSync(localPath, content);
  console.log(`Downloaded s3://${BUCKET}/${s3Key} to ${localPath}`);
}

async function listObjects(prefix: string = ''): Promise<string[]> {
  const command = new ListObjectsV2Command({
    Bucket: BUCKET,
    Prefix: prefix
  });
  const response = await s3Client.send(command);
  return (response.Contents || []).map(obj => obj.Key!);
}

async function deleteObject(s3Key: string): Promise<void> {
  const command = new DeleteObjectCommand({
    Bucket: BUCKET,
    Key: s3Key
  });
  await s3Client.send(command);
  console.log(`Deleted s3://${BUCKET}/${s3Key}`);
}

Working with JSON documents:

import { 
  S3Client, 
  PutObjectCommand, 
  GetObjectCommand 
} from '@aws-sdk/client-s3';
import { Readable } from 'stream';

const s3Client = new S3Client({
  region: process.env.AWS_REGION || 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  }
});

const BUCKET = process.env.S3_BUCKET_NAME!;

async function streamToString(stream: Readable): Promise<string> {
  const chunks: Buffer[] = [];
  for await (const chunk of stream) {
    chunks.push(Buffer.from(chunk));
  }
  return Buffer.concat(chunks).toString('utf-8');
}

async function saveJson<T>(data: T, s3Key: string): Promise<string> {
  const command = new PutObjectCommand({
    Bucket: BUCKET,
    Key: s3Key,
    Body: JSON.stringify(data, null, 2),
    ContentType: 'application/json'
  });
  await s3Client.send(command);
  return `s3://${BUCKET}/${s3Key}`;
}

async function loadJson<T>(s3Key: string): Promise<T> {
  const command = new GetObjectCommand({
    Bucket: BUCKET,
    Key: s3Key
  });
  const response = await s3Client.send(command);
  const content = await streamToString(response.Body as Readable);
  return JSON.parse(content) as T;
}

// Example interfaces for ML workflow
interface TrainingRecord {
  input: number[];
  label: string;
}

interface ModelMetrics {
  accuracy: number;
  f1Score: number;
  trainingTimeSeconds: number;
  hyperparameters: Record<string, number | string>;
}

async function saveTrainingDataset(
  dataset: TrainingRecord[], 
  modelName: string
): Promise<string> {
  const timestamp = new Date().toISOString().replace(/[-:]/g, '').split('.')[0];
  const s3Key = `datasets/${modelName}/${timestamp}/training_data.json`;
  return saveJson(dataset, s3Key);
}

async function saveModelMetrics(
  metrics: ModelMetrics, 
  modelName: string, 
  version: string
): Promise<string> {
  const s3Key = `models/${modelName}/${version}/metrics.json`;
  return saveJson(metrics, s3Key);
}

// Usage
const trainingData: TrainingRecord[] = [
  { input: [1, 2, 3], label: 'A' },
  { input: [4, 5, 6], label: 'B' },
];

const datasetPath = await saveTrainingDataset(trainingData, 'classifier_v1');
console.log(`Dataset saved to ${datasetPath}`);

const metrics: ModelMetrics = {
  accuracy: 0.94,
  f1Score: 0.92,
  trainingTimeSeconds: 3842,
  hyperparameters: { learningRate: 0.001, epochs: 100 }
};

const metricsPath = await saveModelMetrics(metrics, 'classifier_v1', '20240115');
console.log(`Metrics saved to ${metricsPath}`);

A Reusable S3 Storage Class (TypeScript)

import { 
  S3Client, 
  PutObjectCommand, 
  GetObjectCommand,
  DeleteObjectCommand,
  ListObjectsV2Command,
  HeadObjectCommand
} from '@aws-sdk/client-s3';
import { Readable } from 'stream';

interface S3StorageConfig {
  bucket: string;
  prefix?: string;
}

class S3Storage {
  private client: S3Client;
  private bucket: string;
  private prefix: string;

  constructor(config: S3StorageConfig) {
    this.client = new S3Client({
      region: process.env.AWS_REGION || 'us-east-1',
      credentials: {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
      }
    });
    this.bucket = config.bucket;
    this.prefix = config.prefix || '';
  }

  private fullKey(key: string): string {
    return this.prefix ? `${this.prefix}/${key}` : key;
  }

  private async streamToBuffer(stream: Readable): Promise<Buffer> {
    const chunks: Buffer[] = [];
    for await (const chunk of stream) {
      chunks.push(Buffer.from(chunk));
    }
    return Buffer.concat(chunks);
  }

  async put(key: string, data: string | Buffer, contentType?: string): Promise<void> {
    const command = new PutObjectCommand({
      Bucket: this.bucket,
      Key: this.fullKey(key),
      Body: data,
      ContentType: contentType
    });
    await this.client.send(command);
  }

  async putJson<T>(key: string, data: T): Promise<void> {
    await this.put(key, JSON.stringify(data, null, 2), 'application/json');
  }

  async get(key: string): Promise<Buffer> {
    const command = new GetObjectCommand({
      Bucket: this.bucket,
      Key: this.fullKey(key)
    });
    const response = await this.client.send(command);
    return this.streamToBuffer(response.Body as Readable);
  }

  async getJson<T>(key: string): Promise<T> {
    const buffer = await this.get(key);
    return JSON.parse(buffer.toString('utf-8')) as T;
  }

  async exists(key: string): Promise<boolean> {
    try {
      const command = new HeadObjectCommand({
        Bucket: this.bucket,
        Key: this.fullKey(key)
      });
      await this.client.send(command);
      return true;
    } catch {
      return false;
    }
  }

  async delete(key: string): Promise<void> {
    const command = new DeleteObjectCommand({
      Bucket: this.bucket,
      Key: this.fullKey(key)
    });
    await this.client.send(command);
  }

  async list(subPrefix: string = ''): Promise<string[]> {
    const fullPrefix = this.fullKey(subPrefix);
    const command = new ListObjectsV2Command({
      Bucket: this.bucket,
      Prefix: fullPrefix
    });
    const response = await this.client.send(command);
    return (response.Contents || []).map(obj => obj.Key!);
  }
}

// Usage
const storage = new S3Storage({ 
  bucket: process.env.S3_BUCKET_NAME!, 
  prefix: 'ml-experiments' 
});

await storage.putJson('experiment-001/config.json', { 
  learningRate: 0.001, 
  epochs: 100 
});

const config = await storage.getJson<{ learningRate: number; epochs: number }>(
  'experiment-001/config.json'
);

Storage Classes: Saving Money on ML Data

S3 has different storage classes with different pricing. For ML datasets that you access infrequently, you can save serious money:

For ML work, I typically use Standard-IA for datasets and model artifacts. You pay less for storage and the retrieval cost is negligible when you're only pulling data for training runs.

Set the storage class when uploading:

s3.put_object(
    Bucket=BUCKET, 
    Key=s3_key, 
    Body=data, 
    StorageClass='STANDARD_IA'
)
const command = new PutObjectCommand({
  Bucket: BUCKET,
  Key: s3Key,
  Body: data,
  StorageClass: 'STANDARD_IA'
});

Or set up lifecycle rules in the S3 console to automatically transition objects to cheaper storage after a certain number of days.

When Things Go Wrong

"Access Denied": Your IAM policy doesn't grant the right permissions, or the bucket name in your policy doesn't match the actual bucket. Double-check both.

"NoSuchBucket": Bucket name is wrong or the bucket doesn't exist. Remember bucket names are globally unique — maybe someone else has that name.

"NoSuchKey": The object doesn't exist. Check your key path, including any prefixes.

Slow uploads/downloads: S3 is fast, but if you're transferring large files, consider using multipart uploads for files over 100MB. boto3's upload_file handles this automatically.

"InvalidAccessKeyId": Your access key is wrong or has been deactivated. Generate new credentials in IAM.

"SignatureDoesNotMatch": Usually means your secret key is wrong, or there's a clock skew issue (rare in cloud environments).

Organizing Your Bucket

For ML work, I recommend a structure like this:

your-bucket/
├── datasets/
│   ├── project-a/
│   │   ├── 20240115_raw/
│   │   ├── 20240115_processed/
│   │   └── 20240120_augmented/
│   └── project-b/
├── models/
│   ├── project-a/
│   │   ├── v1/
│   │   │   ├── weights.h5
│   │   │   ├── config.json
│   │   │   └── metrics.json
│   │   └── v2/
│   └── project-b/
├── experiments/
│   └── hyperparameter-search-001/
└── configs/
    └── training-defaults.json

Timestamps in folder names. Version numbers. Clear separation between raw data, processed data, and outputs. Your future self will thank you when you need to reproduce results from six months ago.

One More Thing

S3 pricing is usage-based — you pay for storage, requests, and data transfer. For small projects, it's basically free. For large ML datasets, it can add up, but it's still way cheaper than keeping an RDS instance running.

The real cost trap is data transfer out of AWS. Downloading data from S3 to your local machine or to Replit (which is outside AWS) costs money. Uploading is free. If you're moving terabytes around, do the math first.

For most Replit projects, you'll spend more on coffee than on S3. Go build something awesome!

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from Programming

View all →
Programming

What's the Differences Between Relational, NoSQL, Vectors, and Caches

Apr 14, 2026
Programming

Git and Source Control in AI Land

Apr 14, 2026
Programming

Domains, DNS, and why it feels fragile

Apr 14, 2026