Skip to main content
Loading

JSON Format for Aerospike Kafka Source (Outbound) Connector

This document specifies the JSON data serialization format used by the Aerospike Kafka Source (Outbound) Connector. It describes how Aerospike notification messages are represented as JSON objects. Each message represents an event that occurred in an Aerospike database cluster. Different kinds of messages are used to describe events such as record creation/update and record deletions.

Message Formatโ€‹

Each message consists of a JSON object. The type of the message is identified by the value of the msg property. Further properties of the message object are dependent on the message type, which is defined below.

msgMessage TypeDescription
writeWrite MessageRecord creation or update; messages include the record key, record metadata as well as the values of all record bins (post update).
deleteDelete MessageRecord deletion; messages include on the record key.

Write Messageโ€‹

A Write Message object contains the following properties:

KeyFormatNameDescription
msgStringMessage FormatAlways write.
keyArrayRecord KeySee below for definition of Key format. The record key in a Write message always contains the namespace and key digest, and may optionally contain the set name and user key components.
genNumberRecord GenerationA record's generation value changes with each record update.
expNumberExpiration TimeTime when the record will expire, in seconds since the Unix epoch. Zero means the record will not expire.
lutNumberLast-Update TimestampTime when the record was last updated, in milliseconds since the Unix epoch. It is available whenever the Aerospike server ships last-update time. [1]
binsArrayRecord BinsList of record bin values. See below for definition of Bin format.

Note:

[1] When the Aerospike server does not ship lut, Kafka Source (Outbound) connector versions earlier than 4.0.0 ship lut as zero.

Delete Messageโ€‹

A Delete Message object contains the following properties:

KeyFormatNameDescription
msgStringMessage FormatAlways delete.
keyArrayRecord KeySee below for definition of Key format. The record key in a Delete message only contains the namespace and key digest, but not the set name or user key components.
durableBooleanDurable Delete FlagIndicates whether a tombstone was written for the deleted record. See the Durable Deletes documentation for further info.
genNumberRecord GenerationA record's generation value changes with each record update.
lutNumberLast-Update TimestampTime when the record was last updated, in milliseconds since the Unix epoch. It is available whenever Aerospike server ships last-update time.

Key Formatโ€‹

The record key is an array with four elements: Namespace, set name, record digest and user key. Each key contains a namespace and digest; the set name and user key are optional and may be null:

+--------------------+-------------------+-----------------+-------------------------------+
| Namespace [string] | Set [string|null] | Digest [string] | User Key [string|number|null] |
+--------------------+-------------------+-----------------+-------------------------------+
  • Namespace: Aerospike namespace.
  • Set: Optional set name; may be null.
  • Digest: Base64-encoded 160-bit record digest.
  • User Key: The integer, string or bytes value used by the application to uniquely address the record. Will be included in the message only if it is stored on the server. Binary user keys use Base64 encoding.

Bins Formatโ€‹

Each record bin is represented as a JSON object with the following properties:

KeyFormatDescription
nameStringName of the bin
typeStringData type - see below
valuevaries - see belowValue of the bin

Aerospike supports many data types for bin values, including collection data types (CDTs), such as Lists, Maps and Geospatial data (using GeoJSON). The data type of a bin in the JSON representation is specified in the type property of the Bin object. Depending on the specified data type, the value is represented as a JSON string, number, array or object. Depending on the data type, the Bin object may have additional properties.

Bin typeData TypeJSON formatDetails & Additional properties
strStringString
boolBooleanBooleanIntroduced in Aerospike 5.6 and later
intIntegerNumber
floatDoubleNumber
blobBytesStringBase64-encoded
listListArrayordered (boolean): Whether the list is ordered or not
mapMapObjectorder (string, optional): One of key, key-value
geojsonGeoJSONObject

Map and list elements can be of any allowed data type, including nested lists and maps.

Examplesโ€‹

Example "Write"-type messageโ€‹

{
"msg": "write",
"key": ["ns", "set", "YWJjZGVmZ2hpamtsbW5vcHFyc3Q=", null],
"gen": 4,
"exp": 1682797792,
"lut": 1617167159548,
"bins": [
{
"name": "myString",
"type": "str",
"value": "a string value"
},
{
"name": "myBlob",
"type": "blob",
"value": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo="
},
{
"name": "myList",
"type": "list",
"value": ["abc", "def", "ghi", "jkl"],
"ordered": true,
},
{
"name": "myMap",
"type": "map",
"value": {
"i": 42,
"f": 3.1415,
"l": [3, 2, 1, 0]
},
"order": "key-value"
},
{
"name": "myGeo",
"type": "geojson",
"value": { "type": "Point", "coordinates": [1.30824, 103.91327] }
}
]
}

Example "Delete"-type messageโ€‹

{
"msg": "delete",
"key": ["ns", null, "YWJjZGVmZ2hpamtsbW5vcHFyc3Q=", null],
"durable": true,
"gen": 4,
"lut": 1617167159548
}