Skip to main content
Loading

Managing Batch Operations

Batch-Index protocol

A batch is a series of requests that are sent together to the Aerospike database. A batch groups multiple operations into one unit and passes it in a single network trip to each database node. See Batch Operations for more information.

Tuning batches

Background

An incoming batch request from a client is assigned to a specific batch response thread if the status is not "full". A thread is declared "full" when batch-max-buffers-per-queue (255 default) are in use in a queue.

Each batch response thread has a queue of 128KiB buffers. Existing batch requests assigned to a response thread can allocate buffers beyond batch-max-buffers-per-queue, as needed. The “full” designation prevents a queue or response thread from assigning new incoming batch requests. If all response threads are at "full" status, new incoming batch requests are not accepted and an error is returned to the client.

When response threads need a new buffer, they take a buffer from a single pool of unused buffers. This pool is empty at startup and serves all the response threads. Response threads allocate the buffers, and then return them to the pool of unused buffers when done.

A destination client is associated with each 128KB buffer allowing the same response thread to serve multiple batch requests. Excess buffers are destroyed if the number of buffers in the unused pool exceeds batch-max-unused-buffers (default 256). Records that require more than 128KB are allocated a "huge" buffer which is destroyed after use and is not saved in the unused buffer pool.

Operational considerations

The three main configuration options for tuning batch transactions on the server side are the following:

NameDefaultMaxDynamicDescription
batch-max-buffers-per-queue255trueMaximum number of 128KiB response buffers allowed in each batch queue before it is designated "full". Additional buffers beyond batch-max-buffers-per-queue can be allocated for accepted batch requests, if needed. If all batch queues are full, new batch requests are rejected with error code 152 (AS_ERR_BATCH_QUEUES_FULL).
batch-max-unused-buffers256trueMaximum number of 128KiB response buffers allowed in the unused buffer pool for reuse by any response thread. If the limit is reached, new buffers created by response threads at runtime are destroyed upon completion of the batch request. This limits the size of the unused buffer pool that serves all response threads.
batch-index-threadsNumber of CPUs256trueNumber of batch index response worker threads. Each thread has its own queue. These threads only handle returning batch response buffers to the client using sockets. The maximum memory consumption can be computed as: batch-index-threads x batch-max-buffer-per-queue x 128KB. Tuning batch threads to 0 will disable batch functionality, rejecting batch commands with error code 150 (AS_ERR_BATCH_DISABLED)

The default values should be adequate for most workloads, but you may need to adjust those parameters for more demanding workloads to avoid situations where all batch index queues are full, or intermittent CPU spikes. It is also important to understand the memory impact of those parameters.

For example, with 4 index threads and 512 buffers per queue, the total memory used by batch operations is 4 x 512 x 128 KiB = 256 MiB. With 8 index threads and 1024 buffers per queue, the total memory used is 8 x 1024 x 128 KiB = 1 GiB.

The tools package 6.0.x or later is required to use Aerospike Admin (asadm) manage config commands. Otherwise, use the equivalent asinfo - set-config command.

To display the current batch settings, use the show config like batch command in asadm:

Admin> show config like batch
~~~~~~~~~~~~~~~~Service Configuration~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch-index-threads : 4
batch-max-buffers-per-queue: 255
batch-max-requests : 5000
batch-max-unused-buffers : 256
batch-priority : 200
batch-threads : 4
query-batch-size : 100

To monitor your batch statistics, use the asadm command show statistics like batch_index.

In the following example, 62 buffers are created (batch_index_created_buffers) and none are destroyed (batch_index_destroyed_buffers). There are 44 unused buffers (batch_index_unused_buffers) and 18 buffers (batch_index_queue) being used by the 4 index threads.

Admin> show statistics like batch_index
~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch_index_complete : 3534847
batch_index_created_buffers : 62
batch_index_destroyed_buffers: 0
batch_index_error : 46
batch_index_huge_buffers : 0
batch_index_initiate : 3534911
batch_index_queue : 5:5,4:4,4:4,5:5
batch_index_timeout : 0
batch_index_unused_buffers : 44

In the following example, there are 33 unused buffers. In the batch_index_queue, there are 16 + 5 + 5 = 26 buffers in use.

~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch_index_complete : 3520267
batch_index_created_buffers : 62
batch_index_destroyed_buffers: 0
batch_index_error : 44
batch_index_huge_buffers : 0
batch_index_initiate : 3520335
batch_index_queue : 12:16,6:5,0:0,6:5
batch_index_timeout : 0
batch_index_unused_buffers : 33

In the following example, there are 191 unused buffers and the currently active buffers are (165620 - 165324) = 296. The default batch-max-unused-buffers is 256 so there are buffers being created that will be destroyed.

There are also buffers larger than 128 KiB being created to accommodate records larger than 128 KiB. Those are tracked by the batch_index_huge_buffers metric. Huge buffers are destroyed after completed and are not be returned to the buffer pool. The buffer pool can contain a maximum of 256 128 KiB buffers by default. They get used when available and returned to the pool when completed, if the pool size is still below the maximum configured value. If the batch_index_created_buffers and batch_index_destroyed_buffers metrics are incrementing quickly, it is an indication of high garbage collection activity. This may potentially lead to CPU spikes. Tuning the batch-max-unused-buffers configuration option to a higher value may help.

batch_index_complete         :   964670
batch_index_created_buffers : 165620
batch_index_destroyed_buffers: 165324
batch_index_error : 17168
batch_index_huge_buffers : 154797
batch_index_initiate : 1020295
batch_index_queue : 9:23,4:33,11:33,7:6
batch_index_timeout : 38426
batch_index_unused_buffers : 191

If the batch_index_huge_buffers metric is high, you may want to avoid using batch. Large records already saturate socket data buffers in single record mode, so batching them together does not provide any benefit and results in higher memory usage on the server.

To increase the number of idle batch response buffers, use the following asadm command:

asadm -e "enable; manage config service param batch-max-unused-buffers to 512"

You can experiment with the effect of huge buffers on the batch index protocol with the benchmarking application. Try the following benchmarking command:

./run_benchmarks -h 127.0.0.1 -p 3000 -n test -k 10000000 -b 1 -o B:180000 -w RU,99 -g 5000  -T 50 -z 80 -B 10 -t 500000;

Run the command a few times and check your batch metrics:

Admin> show config like batch

batch_index_complete : 258277
batch_index_created_buffers : 11906
batch_index_destroyed_buffers: 11823
batch_index_error : 0
batch_index_huge_buffers : 11823
batch_index_initiate : 258277
batch_index_queue : 0:0,0:0,0:0,0:0
batch_index_timeout : 0
batch_index_unused_buffers : 83

The metrics show that batch_index_huge_buffers and batch_index_destroyed_buffers are the same. This indicates that huge buffers are being created and then destroyed, reducing the efficiency of the batch operations.

Metrics

The server provides the following batch metrics.

NameDescription
batch_index_initiateNumber of batch requests received.
batch_index_queueNumber of batch requests and response buffers remaining on each batch queue. Format: <q1 requests> :<q1 buffers> ,<q2 requests> :<q2 buffers>,...
batch_index_completeNumber of completed batch requests.
batch_index_timeoutNumber of timed-out batch requests.
batch_index_errorNumber of batch requests rejected because of errors.
batch_index_unused_buffersNumber of available 128KB response buffers in the buffer pool.
batch_index_huge_buffersNumber temporary response buffers created that exceeded 128KB. Huge buffers are created when one of the records is retrieved that is greater than 128KB. Huge records do not benefit from batching and can result in excessive memory thrashing on the server.
batch_index_created_buffersNumber of 128KB response buffers created. Response buffers are created when there are no buffers left in the pool. If this number consistently increases and there is available memory, then batch-max-unused-buffers should be increased.
batch_index_destroyed_buffersNumber of 128KB response buffers destroyed. Response buffers are destroyed when there is no slot left to put the buffer back into the pool. The maximum response buffer pool size is batch-max-unused-buffers.
batch-indexBatch performance histogram.

Batch sub transactions

The server also provides the following batch sub transactions for read, writes, deletes, udfs, and (Lua) language. Refer to Metrics Reference.

NameDescription
batch_sub_delete_successNumber of batch delete sub transactions that were completed.
batch_sub_delete_errorNumber of batch delete sub transactions that failed with an error.
batch_sub_delete_timeoutNumber of batch delete sub transactions that timed out.
batch_sub_delete_not_foundNumber of batch delete sub transactions that were not found.
batch_sub_delete_filtered_outNumber of batch delete sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression.
batch_sub_lang_read_successNumber of batch sub language read transactions that were completed.
batch_sub_lang_write_successNumber of batch sub language write transactions that were completed.
batch_sub_lang_delete_successNumber of batch sub language delete transactions that were completed.
batch_sub_lang_errorNumber of batch language sub transactions that failed with an error.
batch_sub_udf_completeNumber of batch UDF sub transactions that were completed.
batch_sub_udf_errorNumber of batch UDF sub transactions that failed with an error.
batch_sub_udf_timeoutNumber of batch UDF sub transactions that timed out.
batch_sub_udf_filtered_outNumber of batch UDF sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression.
batch_sub_write_successNumber of batch write sub transactions that were completed.
batch_sub_write_errorNumber of batch write sub transactions that failed with an error.
batch_sub_write_timeoutNumber of batch write sub transactions that timed out.
batch_sub_write_filtered_outNumber of batch write sub transactions that were filtered out. Transactions filtered out at the bin level by a predicate expression.
batch_sub_proxy_completeNumber of proxied batch sub transactions that completed.
batch_sub_proxy_errorNumber of proxied batch sub transactions that failed with an error.
batch_sub_proxy_timeoutNumber of proxied batch sub transactions that timed out.
batch_sub_read_errorNumber of batch read sub transaction that failed with an error.
batch_sub_read_not_foundNumber of batch read sub transactions that resulted in not found.
batch_sub_read_successNumber of successful batch read sub transactions.
batch_sub_read_timeoutNumber of batch read sub transactions that timed out.
batch_sub_tsvc_errorNumber of batch read sub transactions that failed with an error in the transaction service, before attempting to handle the transaction. For example protocol errors or security permission mismatch.
batch_sub_tsvc_timeoutNumber of batch read sub transactions that timed out in the transaction service, before attempting to handle the transaction. For example, protocol errors or security permission mismatch.
retransmit_all_batch_sub_dup_resObsolete as of Database 6.0 Number of retransmits that occurred during batch sub transactions that were being duplicate resolved. Note this includes retransmits originating on the client as well as proxying nodes.
retransmit_batch_sub_dup_resNumber of retransmits that occurred during batch sub transactions that were being duplicate resolved. Replaced with retransmit_all_batch_sub_dup_res.
retransmit_all_batch_sub_read_dup_resNumber of retransmits that occurred during batch read subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_write_dup_resNumber of retransmits that occurred during batch write subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_delete_dup_resNumber of retransmits that occurred during batch delete subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_udf_dup_resNumber of retransmits that occurred during batch UDF subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_write_repl_write`Number of retransmits that occurred during batch write (insert/update/upsert/replace) subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_delete_repl_write`Number of retransmits that occurred during batch delete subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_udf_repl_write`Number of retransmits that occurred during batch UDF subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_read_repl_ping`Number of retransmits that occurred during SC linearized reads. Includes retransmits originating on the client as well as proxying nodes.
retransmit_all_batch_sub_read_repl_ping`Number of retransmits that occurred during SC linearized read subtransactions within a batch operation. Includes retransmits originating on the client as well as proxying nodes.
early_tsvc_batch_sub_error`Number of errors early in the transaction for batch sub transactions. For example, bad/unknown namespace name or security authentication errors.
from_proxy_batch_sub_delete_successNumber of records successfully deleted by batch sub delete transaction proxied from another node.
from_proxy_batch_sub_delete_errorNumber of records that were not deleted and failed with an error by batch sub delete transaction.
from_proxy_batch_sub_delete_timeoutNumber of records that were not deleted due to time out by batch sub delete transaction.
from_proxy_batch_sub_delete_not_foundNumber of records that were not deleted because that were not found by batch sub transaction.
from_proxy_batch_sub_delete_filtered_outNumber of records that were not deleted because they were filtered out by batch sub transaction. Transactions filtered out at the bin level by a filter expression.
from_proxy_sub_lang_read_successNumber of records that were completed by batch sub language read transactions.
from_proxy_sub_lang_write_successNumber of records that were completed by batch sub language write transactions.
from_proxy_sub_lang_delete_successNumber of records that succeeded by batch sub language delete transactions proxied from another node.
from_proxy_sub_lang_errorNumber of records that failed with an error by batch sub language transaction.
from_proxy_sub_udf_completeNumber of records that were completed by batch sub udf transaction.
from_proxy_sub_udf_errorNumber of records that failed with an error by batch sub udf transaction.
from_proxy_sub_udf_timeoutNumber of records of batch UDF sub transactions proxied from another node that timed out, before attempting to handle this transaction.
from_proxy_sub_udf_filtered_outNumber of records of batch UDF sub transactions proxied from another node that did not happen because the record was filtered out using a filter expression.
from_proxy_sub_write_successNumber of records successfully written by batch sub transactions proxied from another node.
from_proxy_sub_write_errorNumber of batch write sub transactions proxied from another node that failed with an error.
from_proxy_sub_write_timeoutNumber of batch write sub transactions proxied from another node that timed out.
from_proxy_sub_write_filtered_outNumber of batch write sub transactions proxied from another node that did not happen because the record was filtered out using a predicate expression.

Batch log file histograms

Periodically, the Aerospike server writes histograms to the log file. Refer to Latency Monitoring for latency histograms of batch and its sub transactions. In addition, review the Batch Transaction Analysis.

NameDescription
batch_sub_write_masterTime taken for writing all the copies of a record to the master . This only applies for strong consistency enabled namespaces.\n
batch_sub_udf_masterTime taken from partition reserved or after duplicate resolution to an actual master record applied.\n
batch_sub_repl_writeTime taken from the master record written to replica(s) written.\n