Sources Configuration

Sources

Sources are used to import data from some system that allows data loading and export to destinations. For example, if you want to have a replica of data stored at Firebase, you can use Firebase API and store data at some destination.

Collections

As the source may contain inhomogeneous data, it may be split into multiple collections. Each collection represents some subset of data that would be stored to the same structure within the destination (in the case of SQL database, this is a table).

Collections may be static or configurable. For example, Firebase collections are static while Google Analytics report is parametrized (dimensions and metrics).

Full description of the collection:

collections:
- name: "some_name"
type: "collection_type_id"
table_name: "table_name_for_data"
parameters:
field1: "value"
field2: ["values"]
field3:
some_object:
...

Parameter

Description

name (required)

is a unique identifier of collection within a list of collections

type (optional)

determines which data subset must be synchronized. If type absents, type equals to name parameter

table_name (optional)

name of the table to keep synchronized data. If not set, equals to the name of collection

parameters (optional)

if the collection is parametrized, parameter values are set here. A value may be of any type (string, number, boolean, list, object)

If the collection has no parameters, it may be configured only by its name as a string argument. For example:

collections: ["collection1_id", "collection2_id"]

Configuration

Example of source configuration:

sources:
firebase_example_id:
type: firebase
destinations: [ "test_destination" ]
collections: [ "firestore_collection_id" ]
config:
project_id: "firebase_project_id"
key: 'service_account_key_json'
ga_example_id:
type: google_analytics
destinations: [ "test_destination_2" ]
collections:
- name: "report_test"
type: "report"
parameters:
dimensions: [ "ga:country", "ga:yearMonth" ]
metrics: [ "ga:sessions" ]
config:
view_id: "VIEW_ID_VALUE"
auth:
service_account_key: "SERVICE_ACCOUNT_KEY_JSON"
...

Common parameters for all sources (all parameters are required):

Parameter

Description

type

determines the type of a data source from which data would be imported (like google_analytics, firebase)

destinations

list of destination ids where result must be stored

collections

list of collections to synchronize

config

custom parameters for each source type

To see how to configure some type of source, please see the documentation for the exact type.

How it works

Data may be syncronized by time chunks (if data source supports data loading by time intervals) or all data is loaded together. This depends on the type of data source and defined at driver implementation (an entity that loads data). EventNative stores information about synchronized chunks at meta storage (meta storage configuration is described at General Configuration). Time chunk is synchronized if

  • it is not synchronized yet

  • time chunk covers the current moment

  • time chunk covers the previous period to the current one (in case some data is loaded after the period ends)

The result of synchronization is a replica of data from the data source with some enriched fields.

  • eventn_ctx_time_interval field stores information about what synchronization interval

  • eventn_ctx_collection_id contains the type of collection (see documentation on collections below)

  • eventn_ctx_event_id a hash of the synchronized object