Dynamic fetch data from index to generate TreeFlow

Kibana version : 7.9

Elasticsearch version : 7.9

APM Server version : 7.9

APM Agent language and version : java 1.8

Fresh install or upgraded from other version? fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. :-no

I am trying to create a tree flow vega visualization from my APM index. I see from the sample scripts that we need a key and a parent .
My index data is in flat format now and I have to create a hierarchy map . How do I map the index data so that the vega script knows about the key and parent . I am not loading data from json file here . Instead I need to load data dynamically from APM index.

I got the sample script to load data from local json file. But script to load data dynamically and map the key and parent I am unaware.

Is it essential to have a bulk upload of key parent data on to our index to create a map. Can we not map key(example: _source.labels.xxx) and parent(example: _source.labels.yyy) within the vega script itself? Any tips would be appreciated

needed format :

abc ->def ->ghi
-> jkl->mno->pqr
->xyz
-> 123

All the above are different fields from index . How can we achieve this using the index data . Is it essential to have a key parent data structure at index level itself.
Can we not map the key parent in the script itslef with the corresponding fields.

Hi, it will be great if you can share the graph example together with an example of your document structure and the hierarchy you are looking for. It's hard for me to understand exactly how to help you without these info

My data structure is as below

In this structure under the _source ,
labels.OrgId is the parent
for which the child is labels.StoreNum
followed by its child labels.RegisterNum. Further all the data in that index would come as child of labels.RegNum ie., labels.ServiceStatus, labels.ServiceName and few more

I would need a graph as below. But the data needs to e dynamic and followed by the leaf nodes in the below graph would need servicename, service state etc

Thanks for the examples.
One more question: with what you described, looks like a single document identify a single tree path, from the root to the leaf. Have I understood correctly?

Right that's the case with a single document . But when we put time filter multiple docs come in picture and we will need combined tree view from all the documents as data will be common in different docs as well . Probably we will have to iterate through hits.hits and aggregate data based on this . (May be a flatten transform I am not sure ..?)

Hi @markov00 ,

Any inputs on the above. thanks

Hi, I've tried to reproduce a similar data structure and I think I've found a way to implement it:

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 600,
  "height": 600,
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "values": [
        {
          "OrgId": "1001",
          "OrgPlace": "EAST",
          "StoreNum": 101,
          "ServiceStatus": "1"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "EAST",
          "StoreNum": 102,
          "ServiceStatus": "2"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 101
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 102,
          "ServiceStatus": "1"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 102,
          "ServiceStatus": "2"
        }
      ],
      "transform": [
        {
          "type": "fold",
          "fields": [
            "OrgId",
            "OrgPlace",
            "StoreNum",
            "ServiceStatus"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == 'OrgId' ? datum.OrgId : datum.key == 'OrgPlace' ? join([datum.OrgId,datum.OrgPlace],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum, datum.ServiceStatus],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'OrgPlace' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ]
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ]
}

Here a working example

What I've did is using fold to create an new object for every key of your path.
Then I've used two formulas to create a unique name for each node. One formula create the current node id and the other the parentKey.
I've then cleaned up a bit the table filtering out undefined values (basically all the nodes created by the folds without an actual value) and aggregated everything to have a clear hierarchical structure.
Lastly, I've reconstructed the label from the id.

I've applied that to the graph example in vega.

If you want to apply that to Kibana there are few more steps on the data part, where you just have to add your data query and extract the values from the hits.hits array (but this is already shown in the example code of vega in Kibana)

let me know if that works

1 Like

Thank you very much @markov00 . This is what I need. I tried to update the data part as below. It gives me Cannot convert undefined or null to object error.


{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 600,
  "height": 600,
  "padding": 5,
  "signals": [
    {
    
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
     "name" :"table"
         "url": {
      "%context%": true,
      "%timefield%": "@timestamp",
      "index": "apm-*",
      "body": {
        "aggs": {
          "time_buckets": {
            "date_histogram": {
              "field": "@timestamp",
              "interval": {"%autointerval%": true},
              "extended_bounds": {
                "min": {"%timefilter%": "min"},
                "max": {"%timefilter%": "max"}
              },
              "min_doc_count": 0
            }
          }
        },
        "size": 0
     } }
      "format": {"property": "hits.hits[0]"
      } ,
      "transform": [
        {
          "type": "fold",
          "fields": [
            "_source.labels.OrgId",
            "OrgPlace",
            "StoreNum",
            "ServiceStatus"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.labels.OrgId' ? datum._source.labels.OrgId : datum.key == 'OrgPlace' ? join([datum.OrgId,datum.OrgPlace],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum, datum.ServiceStatus],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'OrgPlace' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ]
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ]
}

I have just updated the OrgId part . I think I need to add a null check for org Id. Could you pls check the change that i did is right? Also how can I do a null check here for the fields

I don't think this is right, you need the full array of object so

"format": {"property": "hits.hits" }
1 Like

It's easier if you pick a demo set of documents (like 3/4 docs) and try to fix that in the vega editor online that provides you an easier way to see the data table and what is going on there

How can i test the dynamic data loading part in that though ?

Thanks tried that ..gives me
Infinite extent for field "depth": [Infinity, -Infinity] error ..

can you share just the set of documents to test? I can try locally

1 Like
{
  "_index": "apm-retmon",
  "_type": "_doc",
  "_id": "meOGAXgB-o9SxixKPlyc",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2021-03-05T08:34:38.352049200Z",
    "host.name": "SIMPHONY3",
    "client.geo.location": "41.378197,-81.462354",
    "labels.ServiceName": "Xenv_Application",
    "labels.ServiceSummary": "Xenvironment Engine is down",
    "labels.ServiceStatus": "2",
    "labels.OrgId": "1000",
    "labels.Country": "US",
    "labels.StoreNum": "101",
    "labels.LatLong": "41.378197,-81.462354",
    "labels.RegisterNum": "2",
    "labels.OpType": "Availability",
    "labels.OrgHier": {
      "OrgId": 1000,
      "RegsterNum": "2",
      "Store": "101",
      "District": "1",
      "Region": "West"
    },
    "labels.AppType": "xstore",
    "labels.AlertFlag": "No",
    "labels.StatusChangeTime": "2021-02-05T06:35:13.522Z",
    "labels.Age": "40,439 m",
    "labels.Detail": [
      {
        "Status": "OFFLINE",
        "XenvUIStatus": "OFFLINE",
        "XenvRole": "lead",
        "XenvLeadName": "vm-win10-5",
        "XenvLeadPort": "9096",
        "XenvLeadStatus": "OFFLINE"
      }
    ]
  },
  "fields": {
    "@timestamp": [
      "2021-03-05T08:34:38.352Z"
    ],
    "labels.StatusChangeTime": [
      "2021-02-05T06:35:13.522Z"
    ]
  }
}

{
  "_index": "apm-retmon",
  "_type": "_doc",
  "_id": "mOOGAXgB-o9SxixKEVxG",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2021-03-05T08:34:26.746090200Z",
    "host.name": "SIMPHONY3",
    "client.geo.location": "41.378197,-81.462354",
    "labels.ServiceName": "POS_Payment_Device",
    "labels.ServiceSummary": "PDQ Device is offline",
    "labels.ServiceStatus": "2",
    "labels.OrgId": "1000",
    "labels.Country": "US",
    "labels.StoreNum": "101",
    "labels.LatLong": "41.378197,-81.462354",
    "labels.RegisterNum": "2",
    "labels.OpType": "Availability",
    "labels.OrgHier": {
      "OrgId": 1000,
      "RegsterNum": "2",
      "Store": "101",
      "District": "1",
      "Region": "West"
    },
    "labels.AppType": "xstore",
    "labels.AlertFlag": "No",
    "labels.StatusChangeTime": "2021-02-24T10:57:04.365Z",
    "labels.Age": "12,817 m",
    "labels.Detail": [
      {
        "Status": "OFFLINE",
        "PaymentDeviceIP": "128.100.5.29"
      }
    ]
  },
  "fields": {
    "@timestamp": [
      "2021-03-05T08:34:26.746Z"
    ],
    "labels.StatusChangeTime": [
      "2021-02-24T10:57:04.365Z"
    ]
  },
  "highlight": {
    "labels.AppType": [
      "@kibana-highlighted-field@xstore@/kibana-highlighted-field@"
    ]
  }
}

{
  "_index": "apm-retmon-oracle",
  "_type": "_doc",
  "_id": "PiVWoHgBJTwroTEjClYO",
  "_version": 1,
  "_score": null,
  "_source": {
    "labels.AppType": "OraDb",
    "labels.LatLong": null,
    "StatusChangeTime": "",
    "labels.StoreNum": "CORP",
    "labels.OrgHier": null,
    "labels.OpType": "Availability",
    "ServiceDetail": [
      {
        "LibraryCache": [
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "CMP",
            "PinHits": 472,
            "Pins": 2018,
            "Gets": 2018,
            "GetHits": 472
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "FED APP",
            "PinHits": 968,
            "Pins": 969,
            "Gets": 969,
            "GetHits": 968
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "PDBOPER",
            "PinHits": 0,
            "Pins": 0,
            "Gets": 6,
            "GetHits": 0
          },
       
       
          {
            "InstanceName": "ORCL",
            "Reloads": 4200,
            "Invalidations": 0,
            "NameSpace": "TABLE/PROCEDURE",
            "PinHits": 294879,
            "Pins": 318917,
            "Gets": 136422,
            "GetHits": 124010
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 4031,
            "Invalidations": 3298,
            "NameSpace": "SQL AREA",
            "PinHits": 3813428,
            "Pins": 3850249,
            "Gets": 161629,
            "GetHits": 146974
          }
        ],
        "BufferPoolStatistics": [
          {
            "InstanceName": "ORCL",
            "FreeBufferWait": 0,
            "PhysicalReads": 78823,
            "DbBlockGets": 4243203,
            "DbBlockChange": 4531798,
            "ConsistentGets": 231688190,
            "BufferBusyWait": 428,
            "Name": "DEFAULT",
            "PhysicalWrites": 162190
          }
        ],
        "SysTimeModel": [
          {
            "InstanceName": "ORCL",
            "Value": 938,
            "StatName": "DB CPU"
          },
          {
            "InstanceName": "ORCL",
            "Value": 867712,
            "StatName": "DB time"
          }
        ]
      }
    ],
    "hostname": "DESKTOP-GK9AKCD",
    "@timestamp": "2021-04-05T10:11:36+05:30",
    "client.geo.location": null,
    "ServiceName": "performance",
    "ServiceStatus": "0",
    "labels.Country": null,
    "labels.Age": "",
    "ServiceSummary": "OK  -  Buffer hit ratio: 99.97%, Library cache hit ratio: 98.13%",
    "labels.AlertFlag": "No",
    "labels.OrgId": null,
    "labels.RegisterNum": null
  },
  "fields": {
    "@timestamp": [
      "2021-04-05T04:41:36.000Z"
    ],
    "Memory Usage": [
      0
    ],
    "CPU Status": [
      ""
    ],
    "MemoryUsageStatus": [
      ""
    ],
    "EventTypeTime": [
      null
    ]
  },
  "sort": [
    1617597696000
  ]
}

Above is a sample of various kinds of documents in my index.
There is no labels.region in this doc.You can my be pick labels.Country instead of that

Pls use labels.OrgId, labels.StoreNum, labels.RegisterNum
(these names have a dot operator in it. So I have a challenge in accessing them as well) from the doc.

Pls dont use the ones inside OrgHier.

In the third document certain values are empty /null and thats the expected behaviour. So we need to have null checks for that . pls use the above 3 documents for your testing.
Thanyou @markov00

My current Vega Spec

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": "container",
  "height": "container",
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "format": {
        "property": "hits.hits"
      },
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum['labels.OrgId']"
        },
        {
          "type": "fold",
          "fields": [
            "_source.OrgId",
            "_source.Region",
            "_source.StoreNum",
            "_source.RegisterNum"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.OrgId' ? datum._source.OrgId : datum.key == '_source.Region' ? join([datum._source.labels.OrgId,datum._source.region],'-') : datum.key == '_source.StoreNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum, datum._source.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == '_source.labels.OrgId' ? null : datum.key == '_source.Region' ? datum._source.labels.OrgId : datum.key == '_source.StoreNum' ? join([datum._source.labels.OrgId, datum._source.Region],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ],
      "values": {
        "took": 1,
        "timed_out": false,
        "_shards": {
          "total": 23,
          "successful": 23,
          "skipped": 21,
          "failed": 0
        },
        "hits": {
          "total": 28,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "GzmNoXgBQh5uzqdXvqs2",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T10:22:04.329946300Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Payment_Device",
                "ServiceSummary": "PDQ Device is online",
                "ServiceStatus": "0",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:23:22.231Z",
                "labels.Age": "178",
                "ServiceDetail": [
                  {
                    "Status": "ONLINE",
                    "PaymentDeviceIP": "128.100.5.29"
                  }
                ]
              }
            }
          ]
        },
        "aggregations": {
          "time_buckets": {
            "buckets": []
          }
        }
      }
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

Not looking at the whole picture here but started going down the list off your transformations and the first one doesn't return a result.

Try writing it this to get a value for OrgId

{"type": "formula", "as": "OrgId", "expr": "datum._source['labels.OrgId']"}

1 Like

Thanks @aaron-nimocks .

I have the error Infinite extent for field "depth": [Infinity, -Infinity] now ...Pls assist.

I have tried that and here is my current spec

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": "container",
  "height": "container",
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "format": {
        "property": "hits.hits"
      },
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum._source['labels.OrgId']"
        },
        {
          "type": "formula",
          "as": "Region",
          "expr": "datum._source['labels.Region']"
        },
        {
          "type": "formula",
          "as": "StoreNum",
          "expr": "datum._source['labels.StoreNum']"
        },
        {
          "type": "formula",
          "as": "RegisterNum",
          "expr": "datum._source['labels.RegisterNum']"
        },
        {
          "type": "fold",
          "fields": [
            "_source.OrgId",
            "_source.Region",
            "_source.StoreNum",
            "_source.RegisterNum"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.OrgId' ? datum._source.OrgId : datum.key == '_source.Region' ? join([datum._source.OrgId,datum._source.Region],'-') : datum.key == '_source.StoreNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum, datum._source.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == '_source.OrgId' ? null : datum.key == '_source.Region' ? datum._source.OrgId : datum.key == '_source.StoreNum' ? join([datum._source.OrgId, datum._source.Region],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ],
      "values": {
        "took": 11,
        "timed_out": false,
        "_shards": {
          "total": 23,
          "successful": 23,
          "skipped": 21,
          "failed": 0
        },
        "hits": {
          "total": 2706,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "FyUIoXgBJTwroTEjoWCu",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T07:56:40.636197100Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Application",
                "ServiceSummary": "Xstore Application is down ",
                "ServiceStatus": "2",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:19:31.034Z",
                "labels.Age": "37",
                "ServiceDetail": [
                  {
                    "Xstore": "OFFLINE",
                    "JmxPort": "2020",
                    "ConnectionToPort": "OFFLINE"
                  }
                ]
              },
              "OrgId": "1000",
              "StoreNum": "101",
              "RegisterNum": "2"
            }
          ]
        },
        "aggregations": {
          "time_buckets": {
            "buckets": []
          }
        }
      }
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

There is a bit of confusion about what is the _source datum and the created fields with the formula transformation.
I've cleaned up it a bit and now it looks fine at least on the vega editor online.
I've also slightly changed the way you get the Region as it looks like not always available in the labels object but can be part of the OrgHier object.

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 1000,
  "height": 800,
  "padding": 5,
  "signals": [
    {"name": "labels", "value": true, "bind": {"input": "checkbox"}},
    {
      "name": "layout",
      "value": "tidy",
      "bind": {"input": "radio", "options": ["tidy", "cluster"]}
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": ["line", "curve", "diagonal", "orthogonal"]
      }
    },
    {"name": "separation", "value": false, "bind": {"input": "checkbox"}}
  ],
  "data": [
    {
      "name": "table",
      "format": {"property": "hits.hits"},
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum._source['labels.OrgId']"
        },
        {
          "type": "formula",
          "as": "Region",
          "expr": "datum._source['labels.Region'] || datum._source.OrgHier.Region"
        },
        {
          "type": "formula",
          "as": "StoreNum",
          "expr": "datum._source['labels.StoreNum']"
        },
        {
          "type": "formula",
          "as": "RegisterNum",
          "expr": "datum._source['labels.RegisterNum']"
        },
        {
          "type": "fold",
          "fields": ["OrgId", "Region", "StoreNum", "RegisterNum"]
        }
      ],
      "values": {
        "took": 11,
        "timed_out": false,
        "_shards": {"total": 23, "successful": 23, "skipped": 21, "failed": 0},
        "hits": {
          "total": 2706,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "FyUIoXgBJTwroTEjoWCu",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T07:56:40.636197100Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Application",
                "ServiceSummary": "Xstore Application is down ",
                "ServiceStatus": "2",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:19:31.034Z",
                "labels.Age": "37",
                "ServiceDetail": [
                  {
                    "Xstore": "OFFLINE",
                    "JmxPort": "2020",
                    "ConnectionToPort": "OFFLINE"
                  }
                ]
              },
              "OrgId": "1000",
              "StoreNum": "101",
              "RegisterNum": "2",
              "Region": "West"
            }
          ]
        },
        "aggregations": {"time_buckets": {"buckets": []}}
      }
    },
    {
      "source": "table",
      "name": "transformation",
      "transform": [
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == 'OrgId' ? datum.OrgId : datum.key == 'Region' ? join([datum.OrgId,datum.Region],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.Region, datum.StoreNum],'-') : datum.key == 'RegisterNum' ? join([datum.OrgId, datum.Region, datum.StoreNum, datum.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'Region' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.Region],'-') : datum.key == 'RegisterNum' ? join([datum.OrgId, datum.Region, datum.StoreNum],'-') : null"
        },
        {"type": "filter", "expr": "isDefined(datum.value)"},
        {"type": "aggregate", "groupby": ["id", "parent"]},
        {"type": "formula", "as": "label", "expr": "peek(split(datum.id,'-'))"}
      ]
    },
    {
      "name": "tree",
      "source": "transformation",
      "transform": [
        {"type": "stratify", "key": "id", "parentKey": "parent"},
        {
          "type": "tree",
          "method": {"signal": "layout"},
          "size": [{"signal": "height"}, {"signal": "width - 100"}],
          "separation": {"signal": "separation"},
          "as": ["y", "x", "depth", "children"]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {"type": "treelinks"},
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {"signal": "links"}
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {"scheme": "magma"},
      "domain": {"data": "tree", "field": "depth"},
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {"data": "links"},
      "encode": {
        "update": {"path": {"field": "path"}, "stroke": {"value": "#ccc"}}
      }
    },
    {
      "type": "symbol",
      "from": {"data": "tree"},
      "encode": {
        "enter": {"size": {"value": 100}, "stroke": {"value": "#fff"}},
        "update": {
          "x": {"field": "x"},
          "y": {"field": "y"},
          "fill": {"scale": "color", "field": "depth"}
        }
      }
    },
    {
      "type": "text",
      "from": {"data": "tree"},
      "encode": {
        "enter": {
          "text": {"field": "label"},
          "fontSize": {"value": 9},
          "baseline": {"value": "middle"}
        },
        "update": {
          "x": {"field": "x"},
          "y": {"field": "y"},
          "dx": {"signal": "datum.children ? -7 : 7"},
          "align": {"signal": "datum.children ? 'right' : 'left'"},
          "opacity": {"signal": "labels ? 1 : 0"}
        }
      }
    }
  ],
  "config": {
    "range": {"category": {"scheme": "elastic"}},
    "arc": {"fill": "#54B399"},
    "area": {"fill": "#54B399"},
    "line": {"stroke": "#54B399"},
    "path": {"stroke": "#54B399"},
    "rect": {"fill": "#54B399"},
    "rule": {"stroke": "#54B399"},
    "shape": {"stroke": "#54B399"},
    "symbol": {"fill": "#54B399"},
    "trail": {"fill": "#54B399"},
    "title": {"color": "#343741"},
    "style": {
      "guide-label": {"fill": "#69707d"},
      "guide-title": {"fill": "#343741"},
      "group-title": {"fill": "#343741"},
      "group-subtitle": {"fill": "#343741"}
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {"type": "fit", "contains": "padding"}
}

Thanks.. Regarding the Region .. suppose if we dont have Region at all in a document how can we make it skip that document .. I was able to create the tree flow dynamically . Thanks to the team . Now my question is, if any of the fields are null/empty the script should not fail, that document need to be skipped.

Does the document need to be skipped also if just one field is empty/null?
If so you can also use ES for that, adding an exists query to your existing query Exists query | Elasticsearch Guide [7.12] | Elastic

1 Like