Scaling Namespace Storage for Aerospike on Kubernetes

Scaling namespace storage (vertical scaling) can be a complex topic. The Operator uses Kubernetes StatefulSet for deploying Aerospike clusters. StatefulSet uses PersistentVolumeClaim for providing storage. Currently a PersistentVolumeClaim cannot be updated under these circumstances, which prevents the Operator from providing a simple solution for vertical scaling.

Scale rack-configured storage

We recommend using the Aerospike Rack Awareness feature as a workaround to perform vertical scaling. Instead of changing the size of the existing rack, you create a new, larger rack and delete the old, smaller rack. The Operator automatically transfers the data. In this process, you leave the aerospikeConfig section unmodified and update the storage in the rackConfig section.

Example 1: Rack-configured storage before scaling

For this example, we assume that the cluster is deployed with the custom resource (CR) file aerospike-cluster.yaml. We start with one rack with an id of 1 as the target to replace with a new, larger rack. The cluster currently has two persistentVolume configurations, one with a size of 1Gi and the other with a size of 3Gi.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:7.1.0.0

  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        zone: us-central1-b
        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /dev/sdf
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Block
                  size: 3Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/sdf
.
.
.

To resize /dev/sdf for namespace test, create a new rack inside rackConfig with an updated storage section and delete the old rack from the CR file.

Example 2: Rack-configured storage after scaling

In this example, the second volume is increased from 3Gi to 8Gi.

You can create the new rack in the same physical rack if there is enough space. Use the existing zone/region to hold the new storage and old storage together.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:7.1.0.0

  rackConfig:
    namespaces:
      - test
    racks:
      # Added new rack with id: 2 and removed the old rack  with id: 1
      - id: 2
        zone: us-central1-b
        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /dev/sdf
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Block
                  size: 8Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/sdf
...

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

This creates a new rack with id: 2 and an updated storage section. The Operator removes the old rack after verifying that the Aerospike server has migrated the old data to the new rack.

Check the pods with the get pods command:

> $ kubectl get pods -n aerospike

> NAME                READY   STATUS          RESTARTS   AGE
> aerocluster-2-0     1/1     Running         0          3m6s
> aerocluster-2-1     1/1     Running         0          3m6s
> aerocluster-1-1     1/1     Terminating     0          30s

Scale global storage configurations

In some Kubernetes deployments, you specify all storage as PersistentVolumeClaims and have nothing specified by default in the rackConfig section. This configuration works for normal usage, but it prevents the Aerospike Operator from scaling global storage due to Kubernetes limitations. To scale storage when you have configured your deployment only for global storage, first copy your global storage configuration into the rackConfig section, then make the changes to the CR file as described in the previous section under "Scale rack-configured storage."

Example 1: Global storage before scaling

Example 1 represents the "before" state, prior to scaling. It uses global storage, with example volumes workdir and test in the volumes section under the global storage parameter. Some sections of this CR appear truncated to save space.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike
spec:
  size: 3 
  image: aerospike/aerospike-server-enterprise:6.3.0.2
  storage:
    cleanupThreads: 10
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
      initMethod: dd
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: gp2 
            volumeMode: Filesystem 
            size: 1Gi
      - name: test
        aerospike:
          path: /aerospike/dev/xvdf_test
        source:
          persistentVolume:
            storageClass: openebs-device
            volumeMode: Block 
            size: 60Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret
  podSpec:
    aerospikeInitContainer:
      resources:
        limits:
          cpu: "16"
          memory: 32Gi
        requests:
          cpu: "1"
          memory: 1Gi
    metadata:
      annotations:
        example: annotation
    multiPodPerHost: false
    sidecars:
      - name: exporter
        image: aerospike/aerospike-prometheus-exporter:latest
        ports:
          - containerPort: 9145
            name: exporter
        env:
          - name: "AS_AUTH_USER"
            value: "admin"
          - name: "AS_AUTH_PASSWORD"
            value: "admin123"
          - name: "AS_AUTH_MODE"
            value: "internal"
  rackConfig:
    namespaces:
      - test
      - bar
      - foo
    racks:
      - id: 1
      - id: 2
      - id: 3
  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      ...
    namespaces:
      - name: test
        memory-size: 2000000000
        replication-factor: 2
        nsup-period: 120
        storage-engine:
          type: device
          devices:
            - /aerospike/dev/xvdf_test

To scale the cluster, copy the volumes over to new racks and label them with unique IDs.

Example 2: Global storage after scaling

Example 2 shows a configuration representing the state of the same cluster after scaling. In this example, the volumes are copied over to three racks, labeled with IDs 11, 12, and 13.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 3 
  image: aerospike/aerospike-server-enterprise:6.3.0.2
  storage:
    cleanupThreads: 10
    filesystemVolumePolicy:
      initMethod: deleteFiles
      cascadeDelete: true
    blockVolumePolicy:
      cascadeDelete: true
      initMethod: dd
    volumes:
      - name: workdir
        aerospike:
          path: /opt/aerospike
        source:
          persistentVolume:
            storageClass: gp2 
            volumeMode: Filesystem 
            size: 1Gi
      - name: test
        aerospike:
          path: /aerospike/dev/xvdf_test
        source:
          persistentVolume:
            storageClass: openebs-device
            volumeMode: Block 
            size: 60Gi
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret
  podSpec:
    aerospikeInitContainer:
      resources:
        limits:
          cpu: "16"
          memory: 32Gi
        requests:
          cpu: "1"
          memory: 1Gi
    metadata:
      annotations:
        example: annotation
    multiPodPerHost: false
    sidecars:
      - name: exporter
        image: aerospike/aerospike-prometheus-exporter:latest
        ports:
          - containerPort: 9145
            name: exporter
        env:
          - name: "AS_AUTH_USER"
            value: "admin"
          - name: "AS_AUTH_PASSWORD"
            value: "admin123"
          - name: "AS_AUTH_MODE"
            value: "internal"

  rackConfig:
    namespaces:
      - test
      - bar
      - foo
    racks:
      - id: 11
        storage:
          cleanupThreads: 10
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
            initMethod: dd
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: gp2 
                  volumeMode: Filesystem 
                  size: 1Gi
            - name: test
              aerospike:
                path: /aerospike/dev/xvdf_test
              source:
                persistentVolume:
                  storageClass: openebs-device
                  volumeMode: Block 
                  size: 60Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret
      - id: 12
        storage:
          cleanupThreads: 10
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
            initMethod: dd
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: gp2 
                  volumeMode: Filesystem 
                  size: 1Gi
            - name: test
              aerospike:
                path: /aerospike/dev/xvdf_test
              source:
                persistentVolume:
                  storageClass: openebs-device
                  volumeMode: Block 
                  size: 60Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret
      - id: 13
        storage:
          cleanupThreads: 10
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
            initMethod: dd
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: gp2 
                  volumeMode: Filesystem 
                  size: 1Gi
            - name: test
              aerospike:
                path: /aerospike/dev/xvdf_test
              source:
                persistentVolume:
                  storageClass: openebs-device
                  volumeMode: Block 
                  size: 60Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret
  aerospikeConfig:
    service:
            ...
    security: {}
    network:
            ...
    namespaces:
      - name: test
        memory-size: 2000000000
        replication-factor: 2
        nsup-period: 120
        storage-engine:
          type: device
          devices:
            - /aerospike/dev/xvdf_test
            - /aerospike/dev/xvdf_test_2
      - name: bar
        memory-size: 2000000000
        replication-factor: 2
        nsup-period: 120
        storage-engine:
          type: device
          devices:
            - /aerospike/dev/xvdf_bar
            - /aerospike/dev/xvdf_bar_2
      - name: foo
        memory-size: 2000000000
        replication-factor: 2
        nsup-period: 120
        storage-engine:
          type: device
          devices:
            - /aerospike/dev/xvdf_foo
            - /aerospike/dev/xvdf_foo_2
            - /aerospike/dev/xvdf_foo_3
            - /aerospike/dev/xvdf_foo_4