Managing Strong Consistency clusters

Overview

This page describes how to manage the Aerospike Database cluster nodes of a strong consistency (SC) namespace. The topics here include how to add and remove nodes to the namespace and roster, how to validate partition availability, and how to revive dead partitions.

Add nodes to the cluster and roster

Install and configure Aerospike on the new nodes as described in Configure Strong Consistency.

When the nodes have joined the cluster, use the following command to verify that the result in cluster_size is greater than the result in ns_cluster_size. This is visible in the following output where cluster_size shows 6 rows, while ns_cluster_size shows 5 rows.

Admin> show stat -flip like cluster_size
~Service Statistics (2021-10-22 23:43:35 UTC)~
                    Node|cluster_size
node1.aerospike.com:3000|           6
node2.aerospike.com:3000|           6
node4.aerospike.com:3000|           6
node5.aerospike.com:3000|           6
node6.aerospike.com:3000|           6
node7.aerospike.com:3000|           6
Number of rows: 6

~test Namespace Statistics (2021-10-22 23:43:35 UTC)~
                    Node|ns_cluster_size
node1.aerospike.com:3000|              5
node2.aerospike.com:3000|              5
node4.aerospike.com:3000|              5
node5.aerospike.com:3000|              5
node6.aerospike.com:3000|              5
node7.aerospike.com:3000|              5
Number of rows: 6

Use show roster to see the newly observed nodes in its Observed Nodes section.

note

Tools package 6.2.x or later is required to use the asadm manage roster & show roster commands. Otherwise, use the equivalent asinfo - roster and asinfo - roster-set commands.

Use the following command to copy the Observed Nodes list into the Pending Roster.

Admin> enable
Admin+> manage roster stage observed ns test
Pending roster now contains observed nodes.
Run "manage recluster" for your changes to take affect.

Activate the new roster with the manage recluster command.
```
Admin+> manage recluster
Successfully started recluster
```
Run show roster to confirm that the roster has been updated on all nodes. Verify that the service's cluster_size matches the namespace's ns_cluster_size.

Remove nodes and update the roster

This section describes how to remove a node from an existing namespace configured with SC.

caution

Do not simultaneously remove a number of nodes equal to or greater than your replication-factor (RF). Removing too many nodes simultaneously may result in unavailable partitions, meaning some data is unavailable to the application. If you observe unavailable partitions, re-add the nodes and wait for the cluster to synchronize before proceeding.

Namespaces with replication-factor set to 1 (RF1) have some partitions unavailable whenever any node leaves the cluster, making it impractical to perform a rolling restart or upgrade.

Strong consistency implies the guarantee that with RF N, N copies of data are written to the cluster. A fully formed cluster must contain X nodes where X >= N to satisfy this. Creating a cluster where X = N means that all partitions become unavailable during a single node shut down.

Remove node from the cluster

note

Tools package 6.2.x or later is required to use the asadm manage roster & show roster commands. Otherwise, use the equivalent asinfo - roster and asinfo - roster-set commands.

Run show roster. Verify all roster nodes are present in the cluster.

Admin+> pager on
Admin+> show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:08:54 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         Node            |  Node ID       | Namespace|     Current Roster                                                                           |    Pending Roster                                                                             |                Observed Nodes                                                         
node1.aerospike.com:3000 |BB9070016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
node3.aerospike.com:3000 |BB9060016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
node4.aerospike.com:3000 |BB9050016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
node5.aerospike.com:3000 |BB9040016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
node6.aerospike.com:3000 |BB9010016AE4202 |  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
node7.aerospike.com:3000 |*BB9020016AE4202|  test  |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202 
Number of rows: 6

Shut down safely the nodes to be removed with the following command (in this example on node1.aerospike.com:3000 with node id BB9070016AE4202). Verify you are removing fewer nodes that your configured RF.
note
Any operational procedure that executes a SIGTERM is safe. After startup, a SIGTERM flushes data to disk and properly signal other servers.
```
systemctl stop aerospike
```

When migrations are complete, run the stat command on partitions_remaining until the migrate_partitions_remaining stat becomes zero on all nodes.

Admin> show stat service like partitions_remaining -flip
~~~~~Service Statistics (2021-10-23 00:25:10 UTC)~~~~
                    Node|migrate_partitions_remaining
node2.aerospike.com:3000|                           0
node4.aerospike.com:3000|                           0
node5.aerospike.com:3000|                           0
node6.aerospike.com:3000|                           0
node7.aerospike.com:3000|                           0
Number of rows: 5

Remove node from the roster

Run show roster to verify that BB9070016AE4202 is removed from the pending roster. In the following example there is one fewer node in the Observed Nodes column than in Current Roster and Pending Roster.

Admin+> pager on
Admin+> show roster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Roster (2021-10-23 00:26:56 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    Node|         Node ID|Namespace|                                                                                 Current Roster|                                                                                 Pending Roster|                                                                 Observed Nodes
node1.aerospike.com:3000|BB9070016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
node3.aerospike.com:3000|BB9060016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
node4.aerospike.com:3000|BB9050016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
node5.aerospike.com:3000|BB9040016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
node6.aerospike.com:3000|BB9010016AE4202 |test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
node7.aerospike.com:3000|*BB9020016AE4202|test     |BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202|BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202,BB9010016AE4202
Number of rows: 6

Copy the Observed Nodes list into the Pending Roster.

Admin+> manage roster stage observed ns test
Pending roster now contains observed nodes.
Run "manage recluster" for your changes to take affect.

Run manage recluster to apply the change.

Admin+> manage recluster
Successfully started recluster

Check if startup is complete. Run the following command periodically until it returns ok.
```
asinfo -h [ip of host] -v 'status'
```

caution

Avoid executing a process shut down such as restarting during startup. For more information, see Dead Partitions.

Validate partitions

When you validate partitions, each node reports the global number of dead or unavailable partitions. For example, if the entire cluster has determined that 100 partitions are unavailable, all of the current nodes report 100 unavailable partitions.

Use the following command to display which nodes report dead or unavailable partitions.
```
show stat namespace for <ns> like 'unavailable|dead' -flip
```

In the output of this command, the columns unavailable and dead should be 0 for each node.

Admin> show stat namespace for test like 'unavailable|dead' -flip
~~~~~~test Namespace Statistics (2021-10-23 00:36:43 UTC)~~~~~~~
                    Node|dead_partitions|unavailable_partitions
node1.aerospike.com:30000|              0|                     0
node2.aerospike.com:30000|              0|                     0
node4.aerospike.com:30000|              0|                     0
node5.aerospike.com:30000|              0|                     0
node6.aerospike.com:30000|              0|                     0
Number of rows: 5

Revive dead partitions

You may wish to use your namespace in spite of potentially missing data. For example, you may have entered a maintenance state where you have disabled application use, and are preparing to reapply data from a reliable message queue or other source.

Identify dead partitions.

Admin> show stat namespace for test like dead -flip
~test Namespace Statistics (2021-10-23 00:38:41 UTC)~
                    Node|dead_partitions
node1.aerospike.com:3000|            264
node2.aerospike.com:3000|            264
node4.aerospike.com:3000|            264
node5.aerospike.com:3000|            264
node6.aerospike.com:3000|            264
Number of rows: 5

Run revive to acknowledge the potential data loss on each server.

Admin+> manage revive ns test
~~~Revive Namespace Partitions~~~
                    Node|Response
node1.aerospike.com:3000|ok
node2.aerospike.com:3000|ok
node4.aerospike.com:3000|ok
node5.aerospike.com:3000|ok
node6.aerospike.com:3000|ok
Number of rows: 5    

Run recluster to revive the dead partitions.

Admin+> manage recluster
Successfully started recluster

Verify that there are no longer any dead partitions with the dead_partitions metric.

Admin> show stat namespace for test like dead -flip
~test Namespace Statistics (2021-10-23 00:40:41 UTC)~
                    Node|dead_partitions
node1.aerospike.com:3000|            0
node2.aerospike.com:3000|            0
node4.aerospike.com:3000|            0
node5.aerospike.com:3000|            0
node6.aerospike.com:3000|            0
Number of rows: 5

Overview​

Add nodes to the cluster and roster​

Remove nodes and update the roster​

Remove node from the cluster​

Remove node from the roster​

Validate partitions​

Revive dead partitions​