I am not sure if this is the right forum to discuss problem on plugins behavior. I am using vega plugin for Kibana to generate tree type of visualization. I have data in ES like below -
Let me provide you my example in detail. I have a log file which I read using logstash and put into ElasticSearch. Some sample data stored into ElasticSearch are as below -
{
"audit_message":"Operation_ABC",
"step_id":4,
"@timestamp":"2017-12-01T03:15:22.000Z",
"exec_time":2.457235
},{
"audit_message":"Step_DEF",
"step_id":5,
"@timestamp":"2017-12-01T03:15:27.084Z",
"exec_time":0.020198,
"parent_step_id":4
},{
"audit_message":"Operation_GHI",
"step_id":6,
"@timestamp":"2017-12-01T03:15:32.175Z",
"exec_time":0.015176,
"parent_step_id":5
},{
"audit_message":"Function_JKL",
"step_id":5,
"@timestamp":"2017-12-01T03:17:34.182Z",
"exec_time":4.46E-4,
"parent_step_id":4
},{
"audit_message":"Function_MNO",
"step_id":6,
"@timestamp":"2017-12-01T03:17:39.254Z",
"exec_time":1.56E-4,
"parent_step_id":5
},{
"audit_message":"Operation_PQR",
"step_id":6,
"@timestamp":"2017-12-01T03:17:44.345Z",
"exec_time":0.003522,
"parent_step_id":5
}
In my case, two node's a re linked with parent-child when -
- parent's step_id=(child's step_id)-1 [only this cond is not sufficient since there may multiple step_id with the same value; but a child can not have multiple parents]
So, we have another cond - - parent's @timestamp is the immediate previous @timestamp of the child.
I used stratify transformation on my data to give it to a tree structure. But I am struggling to provide the parentKey in stratify since I can not perform cond#2 as shown above in my vega specification. I tried lookup but, that is only meeting cond#1. Here is what I tried on my primary data source so far -
"transform": [
{
"type": "lookup",
"from": "records", <----- records is secondary datasource. But that is exactly same as my primary datasource
"key": "_source.step_id",
"fields": ["_source.parent_step_id"],
"values": ["_source.@timestamp"],
"as": ["parent_timestamp"],
"default": null
},
{
"type": "stratify",
"key": "_source.@timestamp",
"parentKey": "parent_timestamp"
},
{
"type": "tree",
"method": {"signal": "layout"},
"size": [
{"signal": "height"},
{"signal": "width - 100"}
],
"as": ["y", "x", "depth", "children"]
}
]
This is creating the graph but all the child node's are pointing to one parent; since lookup is only returning the parent on the basis of step_id.
How can I query on the secondary data source with both the conditions as specified above?
