How to migrate from AWS S3 to a compatible provider

Migrating object storage is less scary than it looks: since every provider speaks the same S3 protocol, the bulk of the work isn't rewriting code, it's copying data in an orderly way and switching the endpoint with care. In this guide you'll see a real, reproducible plan to migrate from AWS S3 to a compatible provider with no downtime, with the complete rclone and AWS CLI commands, integrity verification, and the rollback plan in case something goes wrong.

Why S3 migration is simpler than you think

The S3 API is a de facto standard. That means your applications, your backup scripts, and your tools (AWS CLI, rclone, Terraform, Restic, Velero) will keep working the same: the only things that change are the endpoint and the credentials. On OtterStorage the endpoint is https://es-mad-1.s3.otterstorage.io and the API is compatible, so the migration comes down to copying objects from one bucket to another and, when the time comes, pointing your apps at the new target.

The other big reason to migrate is economic. AWS S3 charges for storage, for requests (PUT, GET, LIST), for data egress, and for class-transition operations. OtterStorage charges for storage only: no per-request cost, no per-delete cost, with flat prices of €8/TB·month on HDD, €16 on SSD, and €45 on NVMe. If your S3 bill is dominated by LIST and GET (very typical in backups and data lakes), the savings are immediate. You'll find the numbers on the pricing page and the full reasoning in how to reduce cloud storage costs.

1. Inventory and estimation

Before copying a single byte, you need to know what you have. List all the source buckets, their size, their object count, and whether they use versioning, Object Lock, or lifecycle rules. It's also worth noting which applications write to each bucket: those are the ones you'll have to switch over at the end.

To get the inventory and the total size of a bucket with AWS CLI:

# Size and object count of a bucket
aws s3 ls s3://mi-bucket --recursive --summarize --human-readable | tail -n 2

# Full listing to a file (useful for auditing later)
aws s3api list-objects-v2 --bucket mi-bucket \
  --query 'Contents[].{Key:Key,Size:Size}' --output json > inventario-mi-bucket.json

With the total size you can estimate both the copy time and the AWS egress (data-out) cost, which is billed separately. A quick rule: at an effective 1 Gbps you copy around 400-450 GB/hour. If you have tens of TB, run the copy from a machine with good bandwidth and, ideally, in the same region as your source provider to minimize latency. OtterStorage offers EU-MAD (Madrid), EU-FRA (Frankfurt), and US-EAST regions: pick the one closest to your data and your users.

2. Configure credentials and profiles

Create some access keys in OtterStorage from the console (you can isolate them per bucket, which reduces the blast radius if a key leaks; more detail in access keys isolated per bucket). Then configure two profiles in AWS CLI: one for the AWS source and one for the target.

# ~/.aws/credentials
[aws-origen]
aws_access_key_id = AKIAxxxxxxxxxxxxxxxx
aws_secret_access_key = ********************

[otter]
aws_access_key_id = OTTERxxxxxxxxxxxxxxx
aws_secret_access_key = ********************

And the two equivalent remotes in rclone (~/.config/rclone/rclone.conf):

[aws]
type = s3
provider = AWS
access_key_id = AKIAxxxxxxxxxxxxxxxx
secret_access_key = ********************
region = eu-west-1

[otter]
type = s3
provider = Other
access_key_id = OTTERxxxxxxxxxxxxxxx
secret_access_key = ********************
endpoint = https://es-mad-1.s3.otterstorage.io
region = eu-mad

Check that both sides respond before continuing:

rclone lsd aws:
rclone lsd otter:

3. Create the target bucket

Create the bucket in OtterStorage with the same name (or a new one if you prefer). If the source uses versioning and you want to keep it, enable versioning on the target before copying:

aws s3 mb s3://mi-bucket --endpoint-url https://es-mad-1.s3.otterstorage.io --profile otter

# Enable versioning on the target (if you need it)
aws s3api put-bucket-versioning --bucket mi-bucket \
  --versioning-configuration Status=Enabled \
  --endpoint-url https://es-mad-1.s3.otterstorage.io --profile otter

Step-by-step guide in create a bucket. If your case requires immutability (WORM retention), plan it now: Object Lock must be enabled at bucket creation, not afterward.

4. Initial copy

This is the bulk transfer: all the contents of the source bucket to the target, with the application still writing to the source. You'll do it once and, since it can take hours, it's worth parallelizing well.

Option A: with rclone

rclone is the most convenient tool for bulk migrations: it parallelizes, resumes, and compares hashes. A first full sync:

rclone sync aws:mi-bucket otter:mi-bucket \
  --transfers 32 --checkers 32 \
  --fast-list --s3-chunk-size 64M \
  --progress --log-file=migracion.log

Tune --transfers to your CPU and network. --fast-list reduces the number of LIST requests (irrelevant for cost on OtterStorage, but it helps with AWS rate limiting). Full guide in the rclone documentation.

Option B: with AWS CLI

If you'd rather not install anything else, aws s3 sync makes an equivalent copy. The catch is that a direct bucket-to-bucket copy between two different endpoints isn't trivial, so the usual approach is to go through local disk or a bridge host:

# Download from AWS
aws s3 sync s3://mi-bucket ./staging --profile aws-origen

# Upload to OtterStorage
aws s3 sync ./staging s3://mi-bucket \
  --endpoint-url https://es-mad-1.s3.otterstorage.io --profile otter

If your bridge machine has enough space, this pattern is robust and easy to resume (each sync only moves what's missing). More examples in the AWS CLI guide.

rclone vs AWS CLI: which to use for migrating

In short: for the bulk of the migration, rclone usually wins on speed and verification. For environments where you only have AWS CLI and a host with disk, AWS CLI is perfectly valid.

5. Integrity verification

Don't assume a copy is good without verifying it. There are two levels: count (same number of objects and total size) and checksum (same content object by object).

# Hash verification with rclone (compares MD5/ETag)
rclone check aws:mi-bucket otter:mi-bucket --one-way

# Count and size on each side
rclone size aws:mi-bucket
rclone size otter:mi-bucket

If rclone check finishes with no differences, the copy is faithful. For huge datasets, a sampled verification (downloading and comparing the hash of a percentage of objects) can be enough and much faster. Note any object reported as different and re-sync it individually before continuing.

6. Incremental synchronization

While your application stays in production writing to the source, repeat the sync periodically. Each pass only copies the differences from the previous one, so they'll get faster and faster:

# Incremental pass (only changes since the initial copy)
rclone sync aws:mi-bucket otter:mi-bucket \
  --transfers 32 --checkers 32 --fast-list --progress

The goal is to reach the cutover window with the difference between source and target reduced to minutes of data. Schedule these passes (for example, hourly) in the days leading up to the cutover. Keep in mind one limitation of sync: if an object is deleted at the source, rclone sync will also delete it at the target to keep the mirror; if you'd rather never delete at the target, use rclone copy instead of sync.

7. Cutover window

This is the only sensitive moment and it usually lasts a few minutes. The recommended sequence:

• Pause writes at the source (the app's maintenance mode or freezing the process that writes).
• Run a final incremental sync, which will now be minimal.
• Run a final verification with rclone check.
• Change the endpoint and credentials in your applications' configuration.
• Re-enable writes, now pointing at OtterStorage.

Since the API is identical, switching over is usually as simple as changing two or three environment variables:

export AWS_ENDPOINT_URL=https://es-mad-1.s3.otterstorage.io
export AWS_ACCESS_KEY_ID=OTTERxxxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=********************

In applications that use the AWS SDK, make sure to also set path-style if your client requires it and to check that there are no S3 endpoints hard-wired in the code. After switching over, do real read and write tests from the application itself, not just from the CLI.

8. Rollback plan

A well-executed cutover almost never needs a rollback, but having the plan written down gives you peace of mind. The key: don't delete the source bucket until the new one has been running without incident for days.

• If something fails right after switching, put the old endpoint and credentials back: the source bucket is still intact and up to date (you stopped writing to it minutes ago).
• Keep the migration logs (migracion.log) and the verification result in case you need to reproduce the state.
• Define a grace period (for example, 7-14 days) before the final cleanup.

When you're fully confident, decommission the source bucket on AWS to stop paying for it. On OtterStorage there's no DELETE cost or minimum commitment, so cleaning up leftovers at the target costs you nothing either.

9. Assisted migration with OtterBridge

If you handle a lot of volume, many buckets, or complex cases (full versioning, objects with Object Lock, legal retentions with Legal Hold), OtterBridge is our assisted migration service: we help you plan the inventory, parallelize the copy, and validate integrity, including the metadata and policies that generic tools don't always carry over. Once migrated, you can rely on OtterSync to keep replication across regions and on OtterVault for immutable backups. You'll find the general framework in the documentation.

Frequently asked questions