Restore data from Amazon S3

This page describes how to use the Aerospike tool to restore data from Amazon S3.

Overview

To restore files from Amazon S3, prefix the file and directory names with s3://BUCKET/KEY.

• BUCKET is the name of the source S3 bucket.
• KEY is the key of the object to download/prefix of files in the S3 "directory".
• If using the default S3 endpoint, you must set --s3-region REGION to the region where the bucket is located.
• If using another endpoint, specify that endpoint with --s3-endpoint-override ENDPOINT_URL.
• Files are downloaded in 5MB increments.
• The maximum number of simultaneous asynchronous upload parts across all threads is controlled with --s3-max-async-uploads.

Example

Restore records from an Amazon S3 bucket with the following parameters:

• Source S3 bucket: test-bucket
• Region: us-west-1
• Directory: test-dir

Run the following command:

asrestore -d s3://test-bucket/test-dir --s3-region us-west-1

Required S3 permissions for asrestore

asrestore requires certain permissions to restore from Amazon S3. In the following IAM JSON policy, include the elements listed and replace backup-bucket with the name of the S3 bucket you are using for the restore.

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::backup-bucket/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

S3 restore options

The following options are available for restore from Amazon S3:

Option	Default	Description
--s3-region REGION	-	Sets the S3 region of the bucket being uploaded to/downloaded from. Must be set if using the default S3 endpoint.
--s3-endpoint-override URL	-	Sets the S3 endpoint to use. Must point to an S3-compatible storage system.
--s3-profile PROFILE_NAME	default	Sets the S3 profile to use for credentials.
--s3-max-async-downloads N	32	The maximum number of simultaneous download requests from S3.
--s3-connect-timeout MILLISECONDS	1000	The AWS S3 client's connection timeout in milliseconds. Equivalent to cli-connect-timeout in the AWS CLI, or connectTimeoutMS in the aws-sdk-cpp client configuration.
--s3-log-level LEVEL	Fatal	The log level of the AWS S3 C++ SDK. The possible levels are, from least to most granular. Off Fatal Error Warn Info Debug Trace