ABS Configuration
Aerospike Backup Service (ABS) reads the configuration file aerospike-backup-service.yml to create connections to Aerospike Database namespaces and storage destinations, as well as backup policies, routines, and schedules.
This page explains two sample configuration files included in the ABS GitHub repository and provides details and examples for the configuration parameters.
For the parameter list in OpenAPI format, see the Schemas section in the REST API specification.
Example configuration files
Default Docker Compose ABS Configuration:
The following sample backup service configuration supplied with the Docker Compose stack for ABS consists of four sections:
aerospike-clustersdefines the location and access credentials for ABS to communicate to the Aerospike database, calling that clusterabsCluster1. Since this is a Docker Compose stack, it uses the Aerospike Database Docker container name"aerospike-cluster"as the hostname instead of an IP address.storagedefines the location of the storage for database backups. Here, it creates a storage type calledminioStoragethat uses thes3-endpoint-overrideparameter to send backed up data to MinIO instead of Amazon S3. You can define multiple storage types that can later be used in multiple backup policies.- In
backup-policies, a new policy calledkeepFilesPolicyis defined with simple instructions to run in a single thread and keep all previous backups. A policy is a set of instructions defining how to do a specific type of backup. You can define multiple policies that can be used in various backup routines. backup-routinesspecifies a routine calledminioKeepFilesRoutinethat runs thekeepFilesPolicypolicy daily for full backups and hourly for incremental backups. Routines specify the source cluster to back up data from, a storage type as defined under thestoragesection, and a namespace from the source cluster to back up. You can define multiple routines that can be run according to different schedules or on demand.
aerospike-clusters: absCluster1: seed-nodes: - host-name: "aerospike-cluster" port: 3000 credentials: user: admin password: admin
storage: s3-storage: bucket: my-backup-bucket path: backups s3-profile: default s3-region: eu-central-1 s3-endpoint-override: http://minio:9000
backup-policies: keepFilesPolicy: # Run backup operations in a single thread. parallel: 1 # Previous full backups are not deleted when a new one is created. remove-files: KeepAll
backup-routines: minioKeepFilesRoutine: # 24 hours interval for full backups. interval-cron: "@daily" # 1 hour interval for incremental backups. incr-interval-cron: "@hourly" source-cluster: absCluster1 storage: minioStorage namespace: test backup-policy: keepFilesPolicyDefault Linux ABS Configuration:
The default configuration file supplied with Linux distributions is smaller and simpler than the configuration in the Docker Compose setup.
By default, it sets up a connection to a namespace called "test" in an Aerospike database accessible at 127.0.0.1:3000.
It stores backup files locally at /var/lib/aerospike-backup-service.
aerospike-clusters: cluster1: use-services-alternate: false seed-nodes: - host-name: "127.0.0.1" port: 3000 credentials: user: "admin" password: "admin"
storage: s3-storage: bucket: my-backup-bucket path: backups s3-profile: default s3-region: eu-central-1 s3-endpoint-override: http://minio:9000
backup-policies: policy1: type: 1 parallel: 1
backup-routines: routine1: interval-cron: "@weekly" incr-interval-cron: "@daily" backup-policy: "policy1" source-cluster: "cluster1" storage: "local1" namespace: "test"Configuration options
Search here for individual configuration options. Expand the cards to see more details and/or usage examples.
Options
Connection timeout in milliseconds.
conn-timeout
5000The authentication mode used by the cluster.
INTERNAL, EXTERNAL, PKI
File path with the password string. Mutually exclusive with password field.
aerospike-clusters: cluster1: credentials: user: "admin" password-path: "/path/to/pass.txt"Plaintext password for cluster authentication. If it starts with secrets, path to password stored in Aerospike Secret Agent. Only use this plaintext authentication method for testing, never in production.
aerospike-clusters: cluster1: credentials: user: "admin" password: "admin"Name of secret agent to use to fetch password. Use a previously configured secret agent in the secret-agents configuration stanza.
aerospike-clusters: cluster1: credentials: secret-agent-name: "demoSa"Configuration for one or more secret agents to use to fetch the password.
aerospike-clusters: cluster1: credentials: secret-agent: ... <secret agent config here> ...Plaintext username for cluster authentication.
aerospike-clusters: cluster1: credentials: user: "admin" password: "admin"Authentication details in plain text for the Aerospike cluster.
aerospike-clusters: cluster1: credentials: user: "admin" password: "admin"A user-selected name for the cluster. It is used only in logs and error messages.
Maximum number of simultaneous allowed data reads, or scans, from the cluster during backup. This is a cluster-scope limit for all backup policies to prevent cluster overload.
aerospike-clusters: absDefaultCluster: seed-nodes: - host-name: "localhost" port: 3000 credentials: user: "tester" password: "psw" max-parallel-scans: 8Any non-negative integer
Host name to connect to the seed node for communication with the Aerospike Database cluster.
aerospike-clusters: absCluster1: seed-nodes: - host-name: "aerospike-cluster" port: 3000 credentials: user: admin password: adminPort to connect to the seed node for communication with the Aerospike Database cluster.
aerospike-clusters: absCluster1: seed-nodes: - host-name: "aerospike-cluster" port: 3000 credentials: user: admin password: adminOptional TLS certificate name used for secure connections.
A stanza with connection details of the seed nodes, or the nodes that Aerospike Backup Service uses to communicate with the cluster. On a multi-node cluster, you can supply connection information for a single node and the rest of the nodes will automatically be discovered.
aerospike-clusters: absCluster1: seed-nodes: - host-name: "aerospike-cluster" port: 3000Path to a trusted CA certificate file.
Path to a directory of trusted CA certificates.
Path to the chain file for mutual authentication if the Aerospike cluster supports it.
TLS cipher selection criteria. The format is the same as OpenSSL’s Cipher List Format.
Password to load protected TLS-keyfile (env:VAR, file:PATH, PASSWORD).
Path to the key for mutual authentication if the Aerospike cluster supports it.
Default TLS name used to authenticate each TLS socket connection.
TLS protocol selection criteria. This format is the same as Apache’s SSL Protocol.
A sub-stanza of a particular cluster with details about the cluster TLS configuration.
Whether to use “services-alternate” instead of “services” in info request during cluster tending.
Throttles backup write operations to the backup file(s) to not exceed the given bandwidth in MiB/s.
bandwidth: 10000Compression level to use, or -1 if unspecified.
Compression mode to use. Options are NONE (default) or ZSTD.
`NONE`NONE, ZSTD
Compression details.
Name of the environment variable containing the encryption key.
Path to the file containing the encryption key.
Secret keyword in Aerospike Secret Agent containing the encryption key.
Encryption mode to use. Options are NONE, AES128, or AES256.
`NONE`NONE, AES128, AES256
Controls backup encryption information like the location of an encryption key or the keyword for Aerospike Secret Agent.
File size limit (in MB) for the backup file. If an .asb backup file crosses this size threshold, a new backup file is created.
file-limit: 1024Approximate limit for the number of records to process. Available in Database 4.9 and later.
max-records: 10000Maximum number of retries before aborting the current transaction. Removed in 3.0 and replaced with the retry-policy substanza.
"parallel":1, "remove-files":"KeepAll", "max-retries": 3Only back up record metadata; digest, TTL, generation count, key.
Whether to back up secondary index definitions.
Whether to back up record data, metadata or bin data.
Whether to back up UDF modules.
List of nodes to back up. Formatted as a list of IP addresses and/or host names followed by port numbers. Empty list implies backup of the whole cluster.
node-list: IP_ADDRESS:3000,HOSTNAME:3000IP_ADDRESS:PORT(,IP_ADDRESS:PORT,...) or IP_ADDRESS:TLS_NAME:PORT(,IP_ADDRESS:TLS_NAME:PORT,...)
Maximum number of scan calls to run in parallel. This value should always be equal to or lower than aerospike-clusters.CLUSTER_NAME.max-parallel-scans.
1Limit total returned records per second (RPS). If RPS is zero (the default), the records-per-second limit is not applied.
1000Clear directory or remove output file.
Whether to clear the output directory when a backup is written to it.
KeepAll - do not remove any existing files in the directory
RemoveAll- clear the directory before writing a backup
RemoveIncremental - remove any incremental backups when a full backup is performed, leaving any existing full backups untouched
The total number of full backups to retain. The minimum is 1, meaning each new full backup deletes the previous one. If not specified, all full backups are kept.
full: 5Any integer equal to or greater than 1
The number of most recent full backups for which incremental backups are retained. Cannot exceed the value of full. If omitted, all incremental backups are kept. A value of 0 means that all previous incremental backups will be deleted after each full backup is made.
incremental: 3Integer values equal to or less than 0 and equal to or less than full.
A stanza that optionally specifies retention rules for a given backup policy.
removeFilesPolicy: retention: full: 5 incremental: 3Delay in milliseconds to wait before retrying a failed operation.
500Initial delay between retry attempts, in milliseconds.
Maximum number of retry attempts that will be made. If set to 0, no retries will be performed.
Increases the delay between subsequent retry attempts. The actual delay is calculated as: BaseTimeout * (Multiplier ^ attemptNumber)
Define a policy in this section for the initial waiting time before a retry, number of retries, and a multiplier that extends the wait interval. A retry policy can also be sent as part of a JSON restore request.
backup-policies: example-backup-policy: ... # additional stanzas retry-policy: base-timeout: 1000 # 1000 milliseconds max-retries: 4 multiplier: 2 ... # additional stanzasWhether backup should include keys updated during the backup process. When true, the backup contains only records that last modified before backup started. When false, records updated during backup may be included in the backup.
Socket timeout in milliseconds. If this value is 0, it is set to total-timeout. If both socket-timeout and total-timeout are 0, there is no socket idle time limit.
1000Total socket timeout in milliseconds.
2000Name of the corresponding backup policy.
Example backup routine containing a policy called keepFilesPolicy
backup-routines: minioKeepFilesRoutine: interval-cron: "@daily" incr-interval-cron: "@hourly" source-cluster: absCluster1 storage: minioStorage namespace: test backup-policy: keepFilesPolicyList of bin names to back up. An empty list backs up all bins.
["dataBin"]Interval for incremental backup as a cron expression string. Intervals are extended cron (7 symbols: year, month, week, day, hour, minute, second). Supported reserved words: @yearly, @monthly, @weekly, @daily, @hourly
*/10 * * * * *Interval for full backup as a cron expression string. Intervals are extended cron (7 symbols: year, month, week, day, hour, minute, second). Supported reserved words: @yearly, @monthly, @weekly, @daily, @hourly
0 0 * * * *List of namespaces to back up. Empty list implies backup of whole cluster.
["source-ns1"]List of nodes to back up. Formatted as a list of IP addresses and/or host names followed by port numbers. Empty list implies backup of whole cluster.
IP_ADDRESS:PORT(,IP_ADDRESS:PORT,...)Back up list of partition filters. Partition filters can be ranges, individual partitions, or records after a specific digest within a single partition. Default number of partitions to back up: 0 to 4095: all partitions.
0-1000List of Aerospike Database rack IDs to prefer when reading records for a backup.
[0]The Secret Agent configuration for the routine (optional).
secret-agent-configurationList of set names to back up. An empty list backs up all sets.
["set1"]Name of the corresponding source cluster.
testClusterName of the corresponding storage provider configuration.
Example using minIO storage
storage: minioStorageA top-level stanza specifying the details of one or more backup routines.
Example routine called routine1
backup-routines: routine1: interval-cron: "1/30 * * * * *" incr-interval-cron: "1/5 * * * * *" backup-policy: "policy1" source-cluster: "cluster1" storage: "local1" namespace: "test"A top-level stanza containing cluster objects for each of the Aerospike clusters that ABS is connected to.
aerospike-clusters: cluster1: use-services-alternate: false seed-nodes: - host-name: "127.0.0.1" port: 3000 credentials: user: "admin" password: "admin"Azure storage account key for Shared Key authentication. This is sensitive information. Can be a path in secret agent or an actual value.
string
Azure storage account name for Shared Key authentication.
string
Azure Active Directory client ID for AAD authentication.
string
Azure Active Directory client secret for AAD authentication. This is sensitive information. Can be a path in secret agent or an actual value.
string
Name of the Azure Blob container.
string
Azure Blob service endpoint URL.
string
Root path for the backup repository within the container. If not specified, backups will be saved in the container’s root.
path: backupsstring
Secret Agent configuration (optional). Link to a preconfigured agent. Mutually exclusive with secret-agent.
string
Secret Agent object definition. Mutually exclusive with secret-agent-name.
Azure Active Directory tenant ID for AAD authentication.
string
Stanza defining Azure storage options. Mutually exclusive with other storage objects. Can also take a secret-agent substanza.
storage: azure-blob-storage-example: azure-storage: endpoint: http://127.0.0.1:6000/devstoreaccount1 container-name: testcontainer path: backups account-name: devstoreaccount1GCP storage bucket name.
string
Alternative URL. It is not recommended to use an alternate URL in a production environment.
string
Path to the file containing the service account key in JSON format.
string
Service account key in JSON format. This is sensitive information. Can be a path in secret agent or an actual value.
string
Root path for the backup repository. If not specified, backups will be saved in the bucket’s root.
path: backupsstring
Secret Agent configuration (optional). Link to one of preconfigured agents. Mutually exclusive with secret-agent.
string
Secret Agent object definition. Mutually exclusive with secret-agent-name.
object
Stanza defining GCP storage options. Mutually exclusive with other storage objects. Can also take a secret-agent substanza.
storage: gcp-storage-example: gcp-storage: key-file-path: key-file.json bucket-name: gcp-backup-bucket path: backups endpoint: http://127.0.0.1:9020Root path for the backup directory when backing up locally.
storage: example-local-storage-object: local-storage: path: backupsStanza defining local storage options. Mutually exclusive with other storage objects.
storage: example-local-storage-object: local-storage: path: backupsRoot path for the backup directory.
path: backupsAlternative endpoint for the S3 SDK to communicate (AWS S3 optional).
s3-endpoint-override: http://host.docker.internal:9000Log level of the AWS S3 SDK (AWS S3 optional).
s3-log-level: FATALThe S3 profile name (AWS S3 optional).
s3-profile: defaultS3 region string (AWS S3 optional).
s3-region: eu-central-1Access Key ID for authentication with S3 StaticCredentialsProvider. This is sensitive information. Can be a path in secret agent or an actual value.
string
The S3 bucket name.
string
The maximum number of simultaneous requests allowed from S3.
integer
The minimum size in bytes of individual S3 UploadParts.
integer
The root path for the backup repository within the bucket. If not specified, backups will be saved in the bucket’s root.
path: backupsstring
An alternative endpoint for the S3 SDK to communicate (AWS S3 optional).
s3-endpoint-override: http://host.docker.internal:9000string
The log level of the AWS S3 SDK (AWS S3 optional).
string
The S3 profile name (AWS S3 optional).
string
The S3 region string.
region: eu-central-1string
Secret Access Key for authentication with S3 StaticCredentialsProvider. This is sensitive information. Can be a path in Aerospike Secret Agent or an actual value.
string
Secret Agent configuration (optional). Link to one of preconfigured agents. Mutually exclusive with secret-agent.
string
Secret Agent object definition. Mutually exclusive with secret-agent-name.
Stanza defining AWS S3 storage options. Mutually exclusive with other storage objects. Can also take a secret-agent substanza.
storage: aws-s3-example-object: s3-storage: bucket: as-backup-bucket path: backups s3-region: eu-central-1Type of the storage provider.
type: localA top-level stanza with options for configuring local or remote storage. Define a custom name for each storage object you create, then use further parameters to define each object’s details.
storage: # Example 1: Local Storage storage1: local-storage: path: /local/backups
# Example 2: S3 Storage storage2: s3-storage: bucket: my-backup-bucket path: backups s3-profile: default s3-region: eu-central-1