Skip to main content
Loading

JSON Format for Outbound

JSON Serialization Formatโ€‹

This document specifies the JSON data serialization format used by Aerospike Connect for Kafka. It describes how Aerospike notification messages are represented as JSON objects. Each message represents an event that occurred in an Aerospike database cluster. Different kinds of messages are used to describe events such as record creation/update and record deletions.

JSON (JavaScript Object Notation) is a light-weight data serialization format. It is a text-based format that is easy for humans to read and write, as well as easy for computers to parse and generate. JSON implementations exist for a wide variety of popular programming languages. The official JSON specification can be found at https://json.org/.

JSON Schemaโ€‹

An informal description of the JSON message format can be found in the sections below. Alternatively, a formal definition using JSON Schema is provided in the json-message-schema.json document.

Message Formatโ€‹

Each message consists of a JSON object. The type of the message is identified by the value of the msg property. Further properties of the message object are dependent on the message type, which is defined below.

msgMessage TypeDescription
writeWrite MessageRecord creation or update; messages include the record key, record metadata as well as the values of all record bins (post update).
deleteDelete MessageRecord deletion; messages include on the record key.

Write Messageโ€‹

A Write Message object contains the following properties:

KeyFormatNameDescription
msgStringMessage FormatAlways write.
keyArrayRecord KeySee below for definition of Key format. The record key in a Write message always contains the namespace and key digest, and may optionally contain the set name and user key components.
genNumberRecord GenerationA record's generation value changes with each record update.
expNumberExpiration TimeTime when the record will expire, in seconds since the Unix epoch. Zero means the record will not expire.
lutNumberLast-Update TimestampTime when the record was last updated, in milliseconds since the Unix epoch. It is available whenever the Aerospike server ships last-update time. [1]
binsArrayRecord BinsList of record bin values. See below for definition of Bin format.

Note: [1] When the Aerospike server does not ship lut, then the following versions of these outbound connectors ship lut as zero:

  • JMS outbound connector, versions earlier than 3.0.0
  • Kafka outbound connector, versions earlier than 4.0.0
  • Pulsar outbound connector, versions earlier than 2.0.0

Delete Messageโ€‹

A Delete Message object contains the following properties:

KeyFormatNameDescription
msgStringMessage FormatAlways delete.
keyArrayRecord KeySee below for definition of Key format. The record key in a Delete message only contains the namespace and key digest, but not the set name or user key components.
durableBooleanDurable Delete FlagIndicates whether a tombstone was written for the deleted record. See the Durable Deletes documentation for further info.
genNumberRecord GenerationA record's generation value changes with each record update.
lutNumberLast-Update TimestampTime when the record was last updated, in milliseconds since the Unix epoch. It is available whenever Aerospike server ships last-update time.

Key Formatโ€‹

The record key is an array with four elements: Namespace, set name, record digest and user key. Each key contains a namespace and digest; the set name and user key are optional and may be null:

+--------------------+-------------------+-----------------+-------------------------------+
| Namespace [string] | Set [string|null] | Digest [string] | User Key [string|number|null] |
+--------------------+-------------------+-----------------+-------------------------------+
  • Namespace: Aerospike namespace.
  • Set: Optional set name; may be null.
  • Digest: Base64-encoded 160-bit record digest.
  • User Key: The integer, string or bytes value used by the application to uniquely address the record. Will be included in the message only if it is stored on the server. Binary user keys use Base64 encoding.

Bins Formatโ€‹

Each record bin is represented as a JSON object with the following properties:

KeyFormatDescription
nameStringName of the bin
typeStringData type - see below
valuevaries - see belowValue of the bin

Aerospike supports many data types for bin values, including collection data types (CDTs), such as Lists, Maps and Geospatial data (using GeoJSON). The data type of a bin in the JSON representation is specified in the type property of the Bin object. Depending on the specified data type, the value is represented as a JSON string, number, array or object. Depending on the data type, the Bin object may have additional properties.

Bin typeData TypeJSON formatDetails & Additional properties
strStringString-
boolBooleanBooleanIntroduced in Aerospike 5.6 and later
intIntegerNumber-
floatDoubleNumber-
blobBytesStringBase64-encoded
listListArrayordered (boolean): Whether the list is ordered or not
mapMapObjectorder (string, optional): One of key, key-value
geojsonGeoJSONObject-

Map and list elements can be of any allowed data type, including nested lists and maps.

Examplesโ€‹

Example "Write"-type messageโ€‹

{
"msg": "write",
"key": ["ns", "set", "YWJjZGVmZ2hpamtsbW5vcHFyc3Q=", null],
"gen": 4,
"exp": 1682797792,
"lut": 1617167159548,
"bins": [
{
"name": "myString",
"type": "str",
"value": "a string value"
},
{
"name": "myBlob",
"type": "blob",
"value": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo="
},
{
"name": "myList",
"type": "list",
"value": ["abc", "def", "ghi", "jkl"],
"ordered": true,
},
{
"name": "myMap",
"type": "map",
"value": {
"i": 42,
"f": 3.1415,
"l": [3, 2, 1, 0]
},
"order": "key-value"
},
{
"name": "myGeo",
"type": "geojson",
"value": { "type": "Point", "coordinates": [1.30824, 103.91327] }
}
]
}

Example "Delete"-type messageโ€‹

{
"msg": "delete",
"key": ["ns", null, "YWJjZGVmZ2hpamtsbW5vcHFyc3Q=", null],
"durable": true,
"gen": 4,
"lut": 1617167159548
}

Batch JSON serialization format.โ€‹

If batching is configured then the records in the batch will be a JSON array of JSON formatted records.

Example batch of JSON formatted records.โ€‹

[
{
"msg": "write",
"key": [
"ns",
"set",
"YWJjZGVmZ2hpamtsbW5vcHFyc3Q=",
null
],
"gen": 4,
"exp": 1682797792,
"lut": 1617167159548,
"bins": [
{
"name": "myString",
"type": "str",
"value": "a string value"
},
{
"name": "myBlob",
"type": "blob",
"value": "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo="
},
{
"name": "myList",
"type": "list",
"value": [
"abc",
"def",
"ghi",
"jkl"
],
"ordered": true
},
{
"name": "myMap",
"type": "map",
"value": {
"i": 42,
"f": 3.1415,
"l": [
3,
2,
1,
0
]
},
"order": "key-value"
},
{
"name": "myGeo",
"type": "geojson",
"value": {
"type": "Point",
"coordinates": [
1.30824,
103.91327
]
}
}
]
},
{
"msg": "delete",
"key": [
"ns",
null,
"YWJjZGVmZ2hpamtsbW5vcHFyc3Q=",
null
],
"durable": true,
"gen": 4,
"lut": 1617167159548
}
]

If the keys of a batch are configured to be concatenated, then the payload key of the batch will be a JSON array of JSON formatted keys.

Example batch of JSON formatted keysโ€‹

[
["users", "premium", "k9lDquN7AXrX4BGwwdLiFDwvs30=", "id1234"],
["users", "premium", "JQlDquN7AXrX4BGwwdLiFDwvs30=", "id1235"],
]