Sources are used to import data from some system that allows data loading and export to destinations. For example, if you want to have a replica of data stored at Firebase, you can use Firebase API and store data at some destination.
As the source may contain inhomogeneous data, it may be split into multiple collections. Each collection represents some subset of data that would be stored to the same structure within the destination (in the case of SQL database, this is a table).
Collections may be static or configurable. For example, Firebase collections are static while Google Analytics report is parametrized (dimensions and metrics).
Full description of the collection:
collections:- name: "some_name"type: "collection_type_id"table_name: "table_name_for_data"parameters:field1: "value"field2: ["values"]field3:some_object:...
is a unique identifier of collection within a list of collections
determines which data subset must be synchronized. If type absents, type equals to name parameter
name of the table to keep synchronized data. If not set, equals to the name of collection
if the collection is parametrized, parameter values are set here. A value may be of any type (string, number, boolean, list, object)
If the collection has no parameters, it may be configured only by its name as a string argument. For example:
collections: ["collection1_id", "collection2_id"]
Example of source configuration:
sources:firebase_example_id:type: firebasedestinations: [ "test_destination" ]collections: [ "firestore_collection_id" ]config:project_id: "firebase_project_id"key: 'service_account_key_json'ga_example_id:type: google_analyticsdestinations: [ "test_destination_2" ]collections:- name: "report_test"type: "report"parameters:dimensions: [ "ga:country", "ga:yearMonth" ]metrics: [ "ga:sessions" ]config:view_id: "VIEW_ID_VALUE"auth:service_account_key: "SERVICE_ACCOUNT_KEY_JSON"...
Common parameters for all sources (all parameters are required):
determines the type of a data source from which data would be imported (like google_analytics, firebase)
list of destination ids where result must be stored
list of collections to synchronize
custom parameters for each source type
To see how to configure some type of source, please see the documentation for the exact type.
Data may be syncronized by time chunks (if data source supports data loading by time intervals) or all data is loaded together. This depends on the type of data source and defined at driver implementation (an entity that loads data). EventNative stores information about synchronized chunks at meta storage (meta storage configuration is described at General Configuration). Time chunk is synchronized if
it is not synchronized yet
time chunk covers the current moment
time chunk covers the previous period to the current one (in case some data is loaded after the period ends)
The result of synchronization is a replica of data from the data source with some enriched fields.
eventn_ctx_time_interval field stores information about what synchronization interval
eventn_ctx_collection_id contains the type of collection (see documentation on collections below)
eventn_ctx_event_id a hash of the synchronized object