Upgrading Hardware

This page describes how to upgrade hardware in an Aerospike cluster.

Tip

For best performance, we recommend that all nodes in a cluster have equal storage capacity. If you add a larger SSD to a node, Aerospike cannot utilize the full capacity of that SSD unless you upgrade the other nodes to the same capacity.

When you upgrade your hardware, take down only a single node at a time. When a node comes down, Aerospike distributes data evenly across all nodes in the cluster.

Before you begin

• Review the installation instructions to make sure that your new hardware is configured correctly, as described in Manage storage.
• Determine the correct restart type to ensure that data migrates correctly after the upgrade. If you shut down the node cleanly, you can add the new disk with a fast restart.
• If you are upgrading an entire cluster, verify that no migrations are in progress before you stop a server. Interrupting a cluster during data migration may cause data loss. See Verify data migration status for more information.
• If you add any SSDs, prepare the new drives according to Initialize solid state storage devices.

Upgrade the hardware

1. Stop the server you want to upgrade.

   $ sudo /etc/init.d/aerospike stop

2. Upgrade the hardware.

   Note

   We recommend that you do a full shut down when upgrading hardware, even if your server supports hot swapping for SSDs.

3. Adjust your configuration file to use any expanded memory or SSD storage.

4. Start that server.

   $ sudo /etc/init.d/aerospike start

   Wait for the process to complete and for the server to confirm that the data is ready.

5. Confirm that the server is available.

   Tail the server logs and look for “service ready: soon there will be cake!”.

   $ sudo tail -f /var/log/aerospike/aerospike.log | grep "cake"

Verify data migration status

Verify that no data migration or data rebalancing is in progress before you upgrade a server. After upgrading one server, wait until any migration or rebalancing in the cluster is complete before upgrading another server.

Check the migration status with the following command. When data migration is complete, the Migrates column shows (0,0) for all nodes.

$ asadm
Admin> info

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
               Node               Node                    Ip                  Build   Cluster            Cluster     Cluster         Principal   Client     Uptime
                  .                 Id                     .                      .      Size                Key   Integrity                 .    Conns          .
172.16.146.135:3000   *BB907DF26565000   172.16.146.135:3000                E-3.8.3         1   9203DDDCEBEE7D97   True        BB907DF26565000        1   01:52:04
Number of rows: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                  Node   Avail%   Evictions     Master   Replica     Repl     Stop     Pending        Disk    Disk     HWM        Mem     Mem    HWM      Stop
        .                     .        .           .    Objects   Objects   Factor   Writes    Migrates        Used   Used%   Disk%       Used   Used%   Mem%   Writes%
        .                     .        .           .          .         .        .        .   (tx%,rx%)           .       .       .          .       .      .         .
test        172.16.146.135:3000   99         0.000     82.935 K   0.000     1        false    (0,0)       10.124 MB   1       50      5.062 MB   1       60     90
test                                         0.000     82.935 K   0.000                       (0,0)       10.124 MB                   5.062 MB
Number of rows: 2

Note

Do not upgrade another node in your cluster until the Migrates column shows zeroes. Interrupting data migration may cause data loss.

If you are upgrading a large cluster, you may begin upgrading the next server after a delay of 30-60 seconds, without allowing the data migration to complete.

Reconfigure for additional RAM

When adding more RAM, remember to adjust the high and low water mark settings to ensure sizing is within normal parameters.

Not making proper adjustments can cause unexpected results in available memory percentage.

The following error from /var/log/aerospike/aerospike.log indicates that the node is out of room to write:

Sep 13 2012 22:01:24 GMT: INFO (rw): (base/thr_rw.c:2300) writing pickled failed 8 for digest d480de145a6fac04
Sep 13 2012 22:01:24 GMT: INFO (rw): (base/thr_rw.c:2300) writing pickled failed 8 for digest 55fa14153280be49
Sep 13 2012 22:01:24 GMT: INFO (rw): (base/thr_rw.c:2300) writing pickled failed 8 for digest c9c30cae39037fe5
Sep 13 2012 22:01:24 GMT: INFO (rw): (base/thr_rw.c:2300) writing pickled failed 8 for digest 7f44a33bbc21b2d4
Sep 13 2012 22:01:24 GMT: INFO (rw): (base/thr_rw.c:2300) writing pickled failed 8 for digest b397d7616d0ca1cd