Modeling data relationships in ES and Kibana

Elastic Stack Kibana
1

Hello all,
I have a question about a data modeling in ElasticSearch to fulfill our use case.
We have 2 instances of data - traffic and sale which can be matched by clickout_id field. The relationship is one to many (one traffic clickout can cause multiple sales under same clickout_id) as in this schematic example.
Sometimes, traffic won't contain this "clickout_id" field, sometimes it will and it's a unique value.

one-to-many.png1221×684 11.2 KB

I was revising different approaches, described here, namely:

  1. Nesting objects
  2. Storing data in different _types by providing different mappings within one index
  3. Children-parent relationship

Storing data in different _types, trying to emulate DBMS behaviour won't work, I asked this question earlier. If the field in mappings have the same name, Lucene will not handle it.

My issue is that applying nesting objects approach works fine in Elastic, but will not be working correctly in Kibana, as it does not recognize an array. Thus, I cannot build any visualizations.

The questions are:

  1. Perhaps I did a mistake when uploaded nested documents and/or mappings for specific index (I can describe the process).
  2. Maybe child-parent relationship will suit better, but I cannot see how can I apply this in Kibana
  3. The aim is to query documents in Kibana and Timelion freely

I would appreciate any hints,
Best

2

Just store each event as a document with a transaction ID or similar.

3

That's what we doing right now
But the main questions/possible solutions that are:

  1. Store data in separate indexes unmerged
  2. Merge data and store traffic data and sale merged data in separate indexes
  3. Apply nested objects approach and store subset of sale data within traffic data as an array (does not work out in Kibana)

I know it is hard to recommend something, not being aware of a use case specifics.

4

@warkolm Correct me if I am wrong please, but does Kibana support arrays data?
I saw an issue described here -Github and also a workaround in one discussion .
As far as I understand, it can count the amount of objects in array, but cannot make other aggregations (sum, average, etc.) as in my example - http://prnt.sc/butwkb, right?
Here is the screenshot of how array is displayed in Discover section - http://prntscr.com/bv5m5i