Destination Configuration

Each destination is represented by YAML node of destinations section in the configuration file

destinations:
destination_name1:
type: postgres | snowflake | redshift | s3 | bigquery | clickhouse
mode: stream | batch
only_tokens: [] #optinal, allowed list of tokens. See below
staged: true | false # optional, false by default
data_layout:
table_name_template: events
mappings: #not required - see separate documentation link below
primary_key_fields: #not required - see separate documentation link below
enrichment: #optional - enrichment rules. See below for details
- rule1: #rule 1
- rule2: #rule 1
destination_name2:
....

Except that, destination-specific parameters (usually, connection credentials) should be present. Please, see individual destinations configuration pages (see full list below).

Please set up a destination name (destination_name1 in the example) carefully. This value will be treated as id in multiple places (monitoring counters, logs, folder names) and should not be changed

Following parameters are common for all destination types

Parameter

Value

type

(required)

One of supported destination types

mode

(required)

batch or stream. Use batch for highloaded datasets (>10 events per second) and stream for others

data_layout.table_name_template

(required)

Name of the destination table. Can be either constant string (all events will be written in a single table) or expression in go template language. The subject of expression is the event JSON. Example:data_{{.event_type}}

only_tokens

(required)

List of authorization tokens (secrets) ids. It is used for delimiting data from different tokens to different destinations. Please, see Authorization section. Please, put token id rather then client/server secret

data_layout.mappings

(optional)

Optional parameter to configure the mapping. See Schema and Mappings

data_layout.primary_key_fields

(optional)

Optional parameter to configure primary key (works for PostgresSQL so far). See Primary keys configuration

enrichment

(optional)

Enrichment rules configuration. Every destination can have enrichment rules. see Enrichment Rules page

staged (optional)

If set to true, data may not be stored at the destination. Only dry run operation is supported for staged destinations.

Configuring destinations via HTTP-endpoint

If destinations configuration is generated by an external service, it is possible to externalize via HTTP end-point (or file) as follows:

destinations: 'location'

The location can be http(s):// of a local file (/path/to/file) location and should contain YAML or (JSON that is identical to YAML structure). If the location is an URL, the client will respect If-Modified-Since / Last-Modified caching.

Example of URL content:

{
"destinations": { #json object where inner keys - destinations unique names
"redshift_dab213ibda": { #destination config object
"type": "redshift",
...
},
"clickhouse_in31o31": {
"type": "clickhouse",
...
}
}
}

All supported destinations