Back up data to Amazon S3

This page describes how to back data up to Amazon S3 with the Aerospike backup tool, asbackup.

Overview

To back up files to Amazon S3, run a normal asbackup command and prefix the file and directory names with s3://BUCKET/KEY.

• BUCKET is the name of the target S3 bucket.
• KEY is the key of the object to download/prefix of files in the S3 "directory".
• If using the default S3 endpoint, you must set --s3-region REGION to the region where the bucket is located.
• If using a non-default S3 endpoint, specify that endpoint with --s3-endpoint-override URL.

Files are uploaded in parts asynchronously. --s3-max-async-uploads controls the maximum number of simultaneous asynchronous upload parts across all threads. Each connection to S3 is throttled, so you may need to adjust this number to maximize throughput.

You can upload a maximum of 10,000 parts to S3. Each part must be between 5MB and 5GB, except for the last part which has no lower bound. When backing up to a directory, the value of --file-limit calculates what part size should be used, down to the minimum size of 5MB. The part size is either --file-limit divided by 10,000, or 5MB, whichever is larger.

When backing up to a file, the estimate run before starting the backup is used in the prior calculation to find the proper upload part size.

Example

Back up records to an Amazon S3 bucket with the following parameters:

• Namespace: test
• Target S3 bucket: test-bucket
• Region: us-west-1
• Directory: test-dir

Run the following command:

asbackup -n test -d s3://test-bucket/test-dir --s3-region us-west-1

Required S3 permissions for back up

To configure the required permissions for use with Amazon S3, include the elements listed in the following policy example and replace backup-bucket with the name of the S3 bucket you are using for the backup.

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

S3 backup options

The following options are available for backup to Amazon S3:

Option	Default	Description
--s3-region REGION	-	Sets the S3 region of the bucket being uploaded to or downloaded from. Must be set if using the default S3 endpoint.
--s3-endpoint-override URL	-	Sets the S3 endpoint to use. Must point to an S3-compatible storage system.
--s3-profile PROFILE_NAME	default	Sets the S3 profile to use for credentials.
--s3-min-part-size SIZE IN MEGABYTES	-	An override for the minimum S3 part size to use for file uploads. By default, this size is calculated based on the expected backup file size found either with the value of --file-limit for backup-to-directory or from the backup estimate run before backup-to-file.
--s3-max-async-downloads N	32	Maximum number of simultaneous download requests from S3.
--s3-max-async-uploads N	16	Maximum number of simultaneous upload requests from S3.
--s3-connect-timeout MILLISECONDS	1000	AWS S3 client connection timeout in milliseconds. Equivalent to cli-connect-timeout in the AWS CLI, or connectTimeoutMS in the aws-sdk-cpp client configuration.
--s3-log-level LEVEL	Fatal	Log level of the AWS S3 C++ SDK. The possible levels are, from least to most granular: Off Fatal Error Warn Info Debug Trace