Skip to main content
Loading

Configure Aerospike Connect for Elasticsearch

Configuring streaming from Aerospike to Elasticsearch involves setting up your Aerospike database to send change notifications and modifying the configuration file /etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml.

Set Up Aerospike Database

In your Aerospike database, you must configure Cross-Datacenter Replication (XDR) and enable change notification. You must also configure your Elasticsearch outbound cluster to be a "connector" XDR datacenter and the namespace must point to this as xdr remote datacenter.

See the change notification configuration parameters and example config.

Modify the Config File

You configure streaming from Aerospike to elasticsearch by modifying the /etc/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.yml YAML configuration file.

Configuration levels

There are two levels at which various configuration can be defined:

LevelDescription
RootThis type of property can be configured only once at the root level of the configuration file.
AllThis type of property can be configured at the root level, namespace level and/or at any of the sets' level of that namespace. Generally, this type of configuration is applied to the Aerospike record. The set level configuration takes the higher precedence. If absent, namespace level configuration is used for the record of that particular set or if set is absent. If configuration is also absent at the namespace level, the root level configuration is used for the given record.

Configuration sections

The configuration file has the following sections:

SectionDescriptionConfiguration level
serviceConfigures the connector's listening ports, TLS, and network interface.Root
es-clientConfigures Elasticsearch client properties for the target elasticsearch cluster.Root
doc-idSpecifies how to generate Elasticsearch's document id.All
batchingSpecifies how to batch Aerospike records to send them to Elasticsearch in a single HTTP request.All
bin-transformsSpecifies the bin transformations to apply on the Aerospike record.All
formatSpecifies the message format to use for the outbound messages sent to the Elasticsearch cluster.All
routingConfigures how incoming record updates/deletes from Aerospike are routed to the Elasticsearch cluster.All
namespacesConfigures bin transforms, format, routing at namespace and set level.Root
loggingConfigures the destination and level for the connector's logs.Root
bulk-request-configConfigures the properties of Elasticsearch's Bulk API.All
ignore-error-codesSpecifies a set of error codes to ignore from Elasticsearch.All
caution

If the route is not specified for an Aerospike record, then the record is skipped and logged at the warn level.

Configuration Example

An example Aerospike Connect for Elasticsearch configuration:

service:
address: 0.0.0.0
# Optional TLS section
tls:
port: 8080
mutual-auth: true # For tcp server.
# Mutual auth is triggered only if this property is non-empty in http proxy server.
allowed-peer-names:
- asd.aerospike.com
trust-store:
store-type: PEM
certificate-files: tls/ca.aerospike.com.crt
key-store:
store-type: PEM
store-file: tls/connector.aerospike.com.key.encrypted.pem
store-password-file: tls/storepass
certificate-chain-files: tls/connector.aerospike.com.crt

manage:
address: 0.0.0.0
port: 8902

logging:
file: /var/log/aerospike-elasticsearch-outbound/aerospike-elasticsearch-outbound.log

# Aerospike record routing to an Elasticsearch index.
routing:
mode: static
destination: aerospike

doc-id:
# The digest is used as Elasticsearch's doc-id unless overridden at the child level.
source: digest

es-client:
cluster-config:
type: on-prem
nodes:
- host:
hostname: 192.168.123.234
port: 9200
scheme: https
auth-config:
# We support 3 types of authentication methods which is explained in the other file. Using api-key method here.
type: api-key
api-key-id-file: /tmp/api-key-id-file
api-key-secret-file: /tmp/api-key-secret-file
# Tls config to be used if Elasticsearch is configured to use TLS.
tls-config:
trust-store:
store-file: tls/ca.aerospike.com.truststore.jks
store-password-file: tls/storepass
elasticsearch-request-config:
# There are several other options which are defined in the separate file.
connect-timeout: 10000

batching:
# Overriding one of the batching parameter. Defaults are mentioned in the separate file.
batch-interval: 1000

namespaces:
customTransformer:
batching:
# We are explicitly disabling batching here as we are using class, not a batch-formatter-class. This means that it only
# gets one record as an input. We get multiple records when batching is configured. This disables batching at
# customTransformer namespace unless some specific set overrides it for itself.
enabled: false
format:
mode: custom
class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundCustomFormatter
custom-transformer:
class: com.aerospike.connect.outbound.TestOutboundCustomTransformer
sets:
ignoreErrorCodeSet:
batching:
enabled: false
format:
mode: custom
class: com.aerospike.connect.elasticsearch.outbound.TestElasticsearchOutboundIgnoreErrorCodeCustomFormatter
# We will consider the record to be successfully processed if Elasticsearch returns error code 400.
ignore-error-codes:
- 400

root:
sets:
binTransformsSet:
bin-transforms:
map:
original: transformed
transforms:
- uppercase
docIdBinValueSet:
doc-id:
# Using bin-value type to generate a doc id. The default is digest at top.
source: bin-value
bin-name: es_doc_id
failure-strategy: fail
updateOperationSet:
# Example of bulk-request-config for the update operation. There are more supported config options which are
# defined on a separate page.
bulk-request-config:
ignore-aerospike-delete: true
aerospike-write-operation-mapping:
operation-type: update
doc-as-upsert: true
createOperationSet:
doc-id:
source: static
value: UhP3zQBGIGbdqFIV5pqzdZB6rKA=
# Example for bulk-request-config for create operation. There are more supported config options which are
# defined on a separate page. Create operation means that this will succeed only if the doc doesn't exist in
# Elasticsearch. That's why we have used static doc-id for this set. Static doc-id won't be used practically
# by anyone. This is just for illustrative purpose.
bulk-request-config:
aerospike-write-operation-mapping:
operation-type: create

Mapping between Aerospike and Elasticsearch models

RDBMSAerospikeElasticsearch
DatabaseNamespaceIndex
TableSetIndex (Database 6.0.0+)
RowRecordDocument
ColumnBinFields
PartitionsShards
SchemaMappings
Primary keyPrimary keydocument_id

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.