General Configuration

EventNative is configured through a single YAML file. We follow convention over configuration so the majority of parameters are optional. To get a base file, clone it from GitHub.

Our config file consists of following sections:

  • server — General configuration parameters such as port, authorization, public_url, app logs path, etc.

  • geo — Geo resolution data (extracting city/state information from the IP address). We currently only support MaxMind as a data provider. see Geo Data resolution

  • logEventNative logs all events locally and sends them to their destination in batches. This is where you configure your local temporary path and push frequency.

  • destinations — A set of targets where the final version of events will be stored.

  • synchronization_service — coordination service configuration. It is used in cluster EventNative deployments. see Scaling EventNative

  • notifications — notifier configuration. Server starts, system errors, and panics information will be sent to it. Currently, only Slack notifications are supported.

  • meta.storage - meta storage configuration. At present EventNative supports only Redis. It is used for last events caching. see Events Cache

Example:

server:
name: instance1.domain.com
port: 8081
auth: '193b6281-f211-47a9-b384-102cf4cd2d55'
public_url: https://instances.domain.com
log:
path: /home/eventnative/logs/
metrics.prometheus.enabled: true
geo.maxmind_path: /home/eventnative/app/res/
log:
path: /home/eventnative/logs/events
rotation_min: 5
destinations:
redshift:
bigquery:
synchronization_service:
type: etcd
endpoint: http://your_etcd_host
notifications:
slack:
url: https://slack_web_hook_url
meta:
storage:
redis:
host: redis_host
port: 6379
password: secret_password

Server

All fields from the server section are optional:

Field

Type

Description

Default value

name

string

Unique instance name. It is used in cluster deployments.

unnamed-server

port

int

TCP port for the server to listen on.

8001

auth

objects array/string

see Authorization page.

generated UUID

public_url

string

Service public URL. It is used on the welcome HTML page.

Will be got from Host request header

log.path

string

Path to application logs. If not set, app logs will be in stdout.

-

log.rotation_min

int

Log files rotation minutes. If log.path is configured.

-

auth_reload_sec

int

If an URL is set in auth section, authorization will be reloaded every auth_reload_sec seconds. see Authorization page.

30

destinations_reload_sec

int

If an URL is set in destinations section, destinations will be reloaded every destinations_reload_sec seconds. see Destinations.

40

admin_token

string

see Admin Endpoints page.

-

metrics.prometheus.enabled

boolean

see Application Metrics page.

false

telemetry.disabled.usage

boolean

Flag for disabling telemetry. EventNative collects usage metrics about how you use it and how it is working. We don't collect any customer data.

false

disable_version_reminder

boolean

Flag for disabling log reminder banner about new EventNatvie versions availability.

false

Log

EventNative supports destinations in streaming and batch modes. In the case of batch mode, all events are stored in JSON log files locally to path directory, and every rotation_min minutes they are processed and pushed to destinations. All fields from log section are optional:

Field

Type

Description

Default value

path

string

Events log files path.

/home/eventnative/logs/events

rotation_min

int

Log files rotation minutes.

5

show_in_server

boolean

Flag for debugging. If true - all events JSON data is written in app logs.

false

Destinations

Each config can contain multiple destinations for one instance of EventNative. "Destination" are targets where all events are stored (currently we support RedShift, BiqQuery, Postgres, ClickHouse, Snowflake, and S3). EventNative supports reloadable destinations from an HTTP source, from a local file, and from YAML structure.

HTTP URL

server:
destinations_reload_sec: 40
destinations: https://destinations-source.com/path

Destinations will be reloaded every destinations_reload_sec seconds. Default value is 40.

HTTP requests are sent with If-Modified-Since header. If HTTP response returns 304 code -destinations aren't reconfigured. If destinations content was changed (or logic isn't supported) - HTTP response must return 200 code, Last-Modified header, and body with the following structure:

{
"destinations": { #json object where inner keys - destinations unique names
"redshift_dab213ibda": { #destination config object
"type": "redshift",
....
},
"clickhouse_in31o31": {
"type": "clickhouse",
...
}
}
}

Local file

Destinations file must have the same payload as the HTTP response body above.

server:
destinations_reload_sec: 40
destinations: 'file:///home/eventnative/app/res/destinations.json'

YAML configuration

Destinations can be configured via YAML objects.

destinations:
destination_unique_name_1:
type: destination type
data_layout:
table_template_name: table_name
destination_unique_name_2:
type: destination type
...
postgres: #also unique
mode: stream
data_layout:
table_template_name: table_name

Each destination should have a unique name. If you set your destination name different than the provider name, you need to set up a type (example: destination redshift will have redshift type by default. But if name=redshift_1 type should be explicitly defined as redshift).

Destination objects configuration has the following fields:

Field

Type

Description

Default value

type

string

Type of destination. Available types:

postgres, clickhouse, bigquery, redshift, s3, snowflake.

it is taken from destination name

mode

string

Writing mode. Available modes: batch, stream. In batch mode, incoming events are stored in log files and then are pushed to destination in one transaction. In stream mode, all events are inserted to destinations via persistent queue, where 1 event = 1 transaction.

batch

only_tokens

string array

List of authorization tokens (secrets). It is used for delimiting data from different tokens to different destinations.

all authorization tokens stored by default

data_layout.mapping_type

string

Available types: strict and default. If strict - all event fields which aren't present in mapping rules will be skipped. If default - all event fields will be written in destinations.

default

data_layout.mapping

string array

Rules on how original events should be transformed before sending them to the Data Warehouse see Schema and Mappings page.

-

data_layout.table_name_template

string

Name of the destination table. The name can be either static, or partitioned see Table name selection section.

events

enrichment

object

array

Enrichment rules configuration. Every destination can have 0-N enrichment rules. see Enrichment Rules page

-

break_on_error

boolean

Flag for stopping processing if at least one object in a batch has errors. By default such objects will be skipped and there will be information in app logs.

false

datasource

object

Config object for setting up data source in Postgres/Redshift destinations. see Postgres or Redshift page.

-

s3

object

Config object for setting up S3 access in Redshift/Snowflake/S3 destinations. see S3 page.

-

google

object

Config object for setting up BigQuery destination. see BigQuery page.

-

clickhouse

object

Config object for setting up ClickHouse destination. see ClickHouse page.

-

snowflake

object

Config object for setting up Snowflake destination. see Snowflake page.

-