Backup and Recovery in AWS

AWS provides alternative methods for backing up your data. This document will detail the general methodology and the options available.

Aerospike tools

Using Aerospike tools is the simplest method for backing up and restoring data.

Backup

1. Take a normal backup using asbackup
2. Move the file to a safe location

Restore

Upload the backup file to a system with asrestore. Use asrestore to restore the data back into a cluster.

cat BACKUP.GZ | gunzip | asrestore [...] --input-file -

Benefits

• Simplest
• Cheapest
• Contains SMD data

Drawbacks

• Long duration
• Additional server load while running
• All or nothing across the cluster

EBS snapshots

If you are using persistence, either through EBS directly, or using shadow device(s), you can take advantage of EBS snapshots. EBS snapshots are point-in-time backups of data. This is done at the data block level, transparent to the application (Aerospike).

Backup

To create a backup, simply initiate snapshots on your EBS data volumes.

Note

• EBS snapshots are consistent within a volume, but may not be consistent between volumes. As such, writing to a particular system needs to be stopped entirely, if said system utilizes multiple EBS volumes per namespace. This is due to the (possible) delay between the snapshot events.

• Snapshots do not contain system metadata (SMD). SMD lives under /opt/aerospike/smd and needs to be separately accounted for in a full cluster restore. Individual node restores are unaffected due to SMD being repopulated using other nodes.

• Snapshots are taken quickly, but that does not mean it is immediately available for use. It will take some time before a snapshot is ready to be used.

• Since snapshots are incremental, the first snapshot of any volume will also be the slowest to be ready.

Restore

If your backups were taken using EBS snapshots, you must deploy those snapshots into EBS volumes. After provisioning your replacement EBS volumes, configure the replacement instance exactly as the failed instance had been configured.

This means that EBS volumes are mounted at the same locations and the same number of ephemeral drives are provisioned.

Note

Shadow devices are always treated as authoritative upon cold-start. That means no matter what data may be on the local device, data will always be synchronized from the shadow device back into the local device.

Caution

Be careful when restoring anything short of a full cluster. Due to how Aerospike manages data, there is a risk of records coming back.

Benefits

• Quick
• Minimal impact
• Individual node based

Drawbacks

• More costly
• Additional scripting/coordination required
• Does not account for SMD

Instance failure

If you are using EBS for persistence and your instance fails, you can detach the EBS volume from the failed instance and then attach it to a replacement instance.

1. Remove/Terminate the failed instance, but keep the EBS volume.
2. Deploy a new Instance, with the same private IP as the old instance in the same zone.
3. Attach EBS volume from Step 1 to the replacement instance.
4. Restore the config to the replacement node and restart the node.

Once the node rejoins the cluster, migrations will occur and data will be rebalanced.