Mirror all Amazon S3 buckets with daily cron script using s3cmd

Here’s are a couple of cron scripts to help mirror all your S3 buckets to a local drive and vice versa.

Requires:

    • Amazon S3 account
    • s3cmd installed and configured

From S3 to local folders:

#!/bin/bash

BACKUP_DIR='/backups/S3 Mirror/'
BUCKETS=$(s3cmd ls | grep s3 | awk '{ print $3 }')

for i in $BUCKETS; do
CURRENT_DIR=$BACKUP_DIR${i//s3:\/\//}/
echo Mirroring $i to $CURRENT_DIR
mkdir -p "$CURRENT_DIR"
s3cmd sync --verbose --no-check-md5 --recursive --no-delete-removed $i "$CURRENT_DIR"
done

From local folders to S3:

#!/bin/bash

BACKUP_DIR='/backup/S3 Buckets/'
BUCKETS=$(ls -l "$BACKUP_DIR" | grep '^d' | awk '{ print $8 }')

for i in $BUCKETS; do
CURRENT_DIR=$BACKUP_DIR$i/
S3_DIR=s3://$i
echo Mirroring $CURRENT_DIR to $S3_DIR
s3cmd mb $S3_DIR
s3cmd sync --verbose --no-check-md5 --recursive --no-delete-removed "$CURRENT_DIR" $S3_DIR
done

If you are syncing many/large files, consider using the --no-check-md5 option. To protect against inadvertently losing files, consider using the --no-delete-removed option.