# Dynamic fetch data from index to generate TreeFlow

**URL:** <https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993>\
**Category:** Kibana\
**Tags:** vega\
**Created:** [April 1, 2021, 7:31am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993 "2021-04-01T07:31:08Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 1, 2021, 7:31am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/1 "2021-04-01T07:31:08Z")

</div>

Kibana version : 7.9

Elasticsearch version : 7.9

APM Server version : 7.9

APM Agent language and version : java 1.8

Fresh install or upgraded from other version? fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. :-no

I am trying to create a tree flow vega visualization from my APM index. I see from the sample scripts that we need a key and a parent .  
My index data is in flat format now and I have to create a hierarchy map . How do I map the index data so that the vega script knows about the key and parent . I am not loading data from json file here . Instead I need to load data dynamically from APM index.

I got the sample script to load data from local json file. But script to load data dynamically and map the key and parent I am unaware.

Is it essential to have a bulk upload of key parent data on to our index to create a map. Can we not map key(example: \_source.labels.xxx) and parent(example: \_source.labels.yyy) within the vega script itself? Any tips would be appreciated

needed format :

abc -\>def -\>ghi  
-\> jkl-\>mno-\>pqr  
-\>xyz  
-\> 123

All the above are different fields from index . How can we achieve this using the index data . Is it essential to have a key parent data structure at index level itself.  
Can we not map the key parent in the script itslef with the corresponding fields.

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 1, 2021, 10:15am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/2 "2021-04-01T10:15:27Z")

</div>

Hi, it will be great if you can share the graph example together with an example of your document structure and the hierarchy you are looking for. It's hard for me to understand exactly how to help you without these info

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 1, 2021, 10:25am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/3 "2021-04-01T10:25:13Z")

</div>

> [@markov00](#):
>
> n example of your document structure and the hierarchy you are looking for. It's hard for me to understand exactly how to help you without these info

My data structure is as below

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6ad226b4ef1774a4ea8c4ae197db755a4e85bce4.png)

In this structure under the \_source ,  
labels.OrgId is the parent  
for which the child is labels.StoreNum  
followed by its child labels.RegisterNum. Further all the data in that index would come as child of labels.RegNum ie., labels.ServiceStatus, labels.ServiceName and few more

I would need a graph as below. But the data needs to e dynamic and followed by the leaf nodes in the below graph would need servicename, service state etc

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97c37fbbd071e701773124d2b33b0447720f4673.png)

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 1, 2021, 10:56am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/4 "2021-04-01T10:56:37Z")

</div>

Thanks for the examples.  
One more question: with what you described, looks like a single document identify a single tree path, from the root to the leaf. Have I understood correctly?

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 1, 2021, 11:01am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/5 "2021-04-01T11:01:18Z")

</div>

Right that's the case with a single document . But when we put time filter multiple docs come in picture and we will need combined tree view from all the documents as data will be common in different docs as well . Probably we will have to iterate through hits.hits and aggregate data based on this . (May be a flatten transform I am not sure ..?)

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 2, 2021, 10:42am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/6 "2021-04-02T10:42:37Z")

</div>

Hi @markov00 ,

Any inputs on the above. thanks

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 2, 2021, 1:19pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/7 "2021-04-02T13:19:59Z")

</div>

Hi, I've tried to reproduce a similar data structure and I think I've found a way to implement it:

```auto
{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 600,
  "height": 600,
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "values": [
        {
          "OrgId": "1001",
          "OrgPlace": "EAST",
          "StoreNum": 101,
          "ServiceStatus": "1"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "EAST",
          "StoreNum": 102,
          "ServiceStatus": "2"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 101
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 102,
          "ServiceStatus": "1"
        },
        {
          "OrgId": "1001",
          "OrgPlace": "WEST",
          "StoreNum": 102,
          "ServiceStatus": "2"
        }
      ],
      "transform": [
        {
          "type": "fold",
          "fields": [
            "OrgId",
            "OrgPlace",
            "StoreNum",
            "ServiceStatus"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == 'OrgId' ? datum.OrgId : datum.key == 'OrgPlace' ? join([datum.OrgId,datum.OrgPlace],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum, datum.ServiceStatus],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'OrgPlace' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ]
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ]
}

```

[Here a working example](https://vega.github.io/editor/#/url/vega/N4IgJAzgxgFgpgWwIYgFwhgF0wBwqgegIDc4BzJAOjIEtMYBXAI0poHsDp5kTykSArJQBWENgDsQAGhAATONABONHJnaT0AQXEACOAA8kCHABs4OtgDMdAYSSLMCmkl0mkATzYNMEHZbaKOkg64mzyALQmNOIA1jqyzmSKRhbWMDRwyYqwNFBIJvFImFTSIADuNLL0aABsAAx1MvA0ZFi1DTI4SLIJ4mRoAjIQLeL5EGgA2qCjCHBoIG5McCbjMsT5DHOomIqbMkzRsmig0Tje87BwUDFMbPogAL4PUtNGWwseXpil6yab82pZO5SgdxEdUCdxGdvuhkgk2KU2Kp1ONUBMQIDgTIoH8II5FCAALpPF4gGbvKKxVYgX7-dAJJBkCT5EGHY4gU7ndAQZZXb4yJFqCSo9GUubYhiKUilBlM0YmREOGBsOUs4nPV6zeY8rrJIWSNYbLaWMZwfZsiEcqFckCXa63e5PQkyWRFFBozXvYpMMw-I0i0AAeUUZAAkuCQABGBqR0rBsgABTcUHeAFFNABlAAqpQzmACcAAcgwEGho5GZBnMsRcnA80UGKio49SfHw-No3VYzJ40mkCn5uns7n84oiyWy3UAEyV6u1+uYRvzKctoMh9voTvdkC95PvADqqeHldH49LqHLGp364jW7jIb7A-Qh+PIDzBeL5+jM7fc5TC6XTdV2vMNbxje9Ez3eYXxzE8PwnC9p1nKV52KRcmxXB5nQxZJxAgfxFHPKYMXcHB3n8EwjhkSwMkokUQPbHsHyguCx0-XM-zrNCl2JUlMFI8iAgQBg3FKJAm0qUoDBwAl6QbBBKBiOB3B0ABeVSdAAcjbWRNJ0AB+QpFwUnSdFQIyS0U5S1I07TmP7OA9MM4Q2GiAAKCZXWMygdKkLzLN3BznU08JNIASjMiyFKUlT1K0982JLJydBc9zPPknybykKLMsghzsv8hSErPYLQoi8zCqs2LbKrFD-24iBktS8QPMq3ycsClMCoy4rP267zapreqGwgUrwsi8QRIVK9+LI+YCOE0SZHE+ZdTgcR+RAaTZLkDKYpsrSdOSyaTAKCq9usuK7LylNkram9Isq-art6pKDJS1yWvS7z2vum64DG8qcuemrOIAxr3ua1qMt+mH7K6nLXoQQGJqm4DZvImgTHxKT9Bk+YaAgAAROAaPEOBZDcyraTgML0YE+ZGSSPhHFKJIvBwJhgTRDkqJANaNqJGaGfQBaRJQZam0WZZcfx9AyLgGI3IgUw6CpjLKikELwrprCr3JAEx3FEAxElJ8MSQH1jZ2Fx8KEyZQAx7UbbUSwsRAGKCb5gXMAAaWU1b7HW75hbm9AdjgY3ZnoMJ2WGMh5XmNxPHOZ4TZoAAvLZiPjxP0GaVoQ9JXOWXQCoqhgHRwh0TtHmwnV7CKdQ45GUuTbgXUm4kFsQBWnn3fuF0O+qbF0kosdJGJXjPST6IYmpU3si9I3ShtvCFodkiw5wyPKXn+nt73roR5AAIMkF-Oz4ziRigVIYYCQMPQBLhV0D38YnSwoY8jMAMyTeC4bATABFXiLBY0Q4D2FKLhMgWxn6XC1OgZAZBkA91kGwZA0R2ReXdDvY2NFlgRnkKoGAPcs6KARNsXYcAv4gGQIofeHot7vGPqQ6iFDzygBwbPKkPd1pQDCHAkADAcBeSEaw9kBDKKB2qGnPEFClLshpvMAAxFAdRjwnih3eBAdwCBbh3xAJYDh2C3SG0jnw8QAj5DsmDpkFuWclFGknHUOROw2CKMtMo9AKjLB+M0TIERYj2T3EtFIiMjoZDc1AOE+YwI040VOnHH+7wBHAIJNRWiRDh6kK0dogEBhNrGIwaY4o5i5hp34YI2xG17GWkcPoGEMSslJ0tjLBJN8MyZyEd4gAnGnJg4llgQKcX8d4CBKiyF9CSYRoiihCNCc0wh8xIkgGiUYlp6B4kukWenBObdKo5HHutd64QADskUzk93yCMFu+zX67W8kc2QE93qaWUIXPS5lNJmEsJgTSPckT9joOsl+rSlgrHepGSKdRNGfyAA)

What I've did is using `fold` to create an new object for every key of your path.  
Then I've used two `formula`s to create a unique name for each node. One formula create the current node id and the other the `parentKey`.  
I've then cleaned up a bit the table filtering out undefined values (basically all the nodes created by the folds without an actual value) and aggregated everything to have a clear hierarchical structure.  
Lastly, I've reconstructed the `label` from the id.

I've applied that to the graph example in vega.

If you want to apply that to Kibana there are few more steps on the data part, where you just have to add your data query and extract the values from the `hits.hits` array (but this is already shown in the example code of vega in Kibana)

let me know if that works

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 2, 2021, 3:19pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/8 "2021-04-02T15:19:34Z")

</div>

> [@markov00](#):
>
> ` "name": "table",`

Thank you very much @markov00 . This is what I need. I tried to update the data part as below. It gives me Cannot convert undefined or null to object error.

```auto

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 600,
  "height": 600,
  "padding": 5,
  "signals": [
    {
    
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
     "name" :"table"
         "url": {
      "%context%": true,
      "%timefield%": "@timestamp",
      "index": "apm-*",
      "body": {
        "aggs": {
          "time_buckets": {
            "date_histogram": {
              "field": "@timestamp",
              "interval": {"%autointerval%": true},
              "extended_bounds": {
                "min": {"%timefilter%": "min"},
                "max": {"%timefilter%": "max"}
              },
              "min_doc_count": 0
            }
          }
        },
        "size": 0
     } }
      "format": {"property": "hits.hits[0]"
      } ,
      "transform": [
        {
          "type": "fold",
          "fields": [
            "_source.labels.OrgId",
            "OrgPlace",
            "StoreNum",
            "ServiceStatus"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.labels.OrgId' ? datum._source.labels.OrgId : datum.key == 'OrgPlace' ? join([datum.OrgId,datum.OrgPlace],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum, datum.ServiceStatus],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'OrgPlace' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.OrgPlace],'-') : datum.key == 'ServiceStatus' ? join([datum.OrgId, datum.OrgPlace, datum.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ]
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ]
}

```

I have just updated the OrgId part . I think I need to add a null check for org Id. Could you pls check the change that i did is right? Also how can I do a null check here for the fields

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 2, 2021, 3:23pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/9 "2021-04-02T15:23:58Z")

</div>

> [@anjana1](#):
>
> ```auto
> "format": {"property": "hits.hits[0]"
> }
> 
> ```

I don't think this is right, you need the full array of object so

```auto
"format": {"property": "hits.hits" }

```

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 2, 2021, 3:25pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/10 "2021-04-02T15:25:29Z")

</div>

It's easier if you pick a demo set of documents (like 3/4 docs) and try to fix that in the vega editor online that provides you an easier way to see the data table and what is going on there

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 2, 2021, 3:38pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/11 "2021-04-02T15:38:41Z")

</div>

How can i test the dynamic data loading part in that though ?

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 2, 2021, 3:39pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/12 "2021-04-02T15:39:21Z")

</div>

Thanks tried that ..gives me  
Infinite extent for field "depth": [Infinity, -Infinity] error ..

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 2, 2021, 3:50pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/13 "2021-04-02T15:50:55Z")

</div>

can you share just the set of documents to test? I can try locally

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 5, 2021, 3:52am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/14 "2021-04-05T03:52:03Z")

</div>

```auto
{
  "_index": "apm-retmon",
  "_type": "_doc",
  "_id": "meOGAXgB-o9SxixKPlyc",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2021-03-05T08:34:38.352049200Z",
    "host.name": "SIMPHONY3",
    "client.geo.location": "41.378197,-81.462354",
    "labels.ServiceName": "Xenv_Application",
    "labels.ServiceSummary": "Xenvironment Engine is down",
    "labels.ServiceStatus": "2",
    "labels.OrgId": "1000",
    "labels.Country": "US",
    "labels.StoreNum": "101",
    "labels.LatLong": "41.378197,-81.462354",
    "labels.RegisterNum": "2",
    "labels.OpType": "Availability",
    "labels.OrgHier": {
      "OrgId": 1000,
      "RegsterNum": "2",
      "Store": "101",
      "District": "1",
      "Region": "West"
    },
    "labels.AppType": "xstore",
    "labels.AlertFlag": "No",
    "labels.StatusChangeTime": "2021-02-05T06:35:13.522Z",
    "labels.Age": "40,439 m",
    "labels.Detail": [
      {
        "Status": "OFFLINE",
        "XenvUIStatus": "OFFLINE",
        "XenvRole": "lead",
        "XenvLeadName": "vm-win10-5",
        "XenvLeadPort": "9096",
        "XenvLeadStatus": "OFFLINE"
      }
    ]
  },
  "fields": {
    "@timestamp": [
      "2021-03-05T08:34:38.352Z"
    ],
    "labels.StatusChangeTime": [
      "2021-02-05T06:35:13.522Z"
    ]
  }
}

{
  "_index": "apm-retmon",
  "_type": "_doc",
  "_id": "mOOGAXgB-o9SxixKEVxG",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2021-03-05T08:34:26.746090200Z",
    "host.name": "SIMPHONY3",
    "client.geo.location": "41.378197,-81.462354",
    "labels.ServiceName": "POS_Payment_Device",
    "labels.ServiceSummary": "PDQ Device is offline",
    "labels.ServiceStatus": "2",
    "labels.OrgId": "1000",
    "labels.Country": "US",
    "labels.StoreNum": "101",
    "labels.LatLong": "41.378197,-81.462354",
    "labels.RegisterNum": "2",
    "labels.OpType": "Availability",
    "labels.OrgHier": {
      "OrgId": 1000,
      "RegsterNum": "2",
      "Store": "101",
      "District": "1",
      "Region": "West"
    },
    "labels.AppType": "xstore",
    "labels.AlertFlag": "No",
    "labels.StatusChangeTime": "2021-02-24T10:57:04.365Z",
    "labels.Age": "12,817 m",
    "labels.Detail": [
      {
        "Status": "OFFLINE",
        "PaymentDeviceIP": "128.100.5.29"
      }
    ]
  },
  "fields": {
    "@timestamp": [
      "2021-03-05T08:34:26.746Z"
    ],
    "labels.StatusChangeTime": [
      "2021-02-24T10:57:04.365Z"
    ]
  },
  "highlight": {
    "labels.AppType": [
      "@kibana-highlighted-field@xstore@/kibana-highlighted-field@"
    ]
  }
}

{
  "_index": "apm-retmon-oracle",
  "_type": "_doc",
  "_id": "PiVWoHgBJTwroTEjClYO",
  "_version": 1,
  "_score": null,
  "_source": {
    "labels.AppType": "OraDb",
    "labels.LatLong": null,
    "StatusChangeTime": "",
    "labels.StoreNum": "CORP",
    "labels.OrgHier": null,
    "labels.OpType": "Availability",
    "ServiceDetail": [
      {
        "LibraryCache": [
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "CMP",
            "PinHits": 472,
            "Pins": 2018,
            "Gets": 2018,
            "GetHits": 472
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "FED APP",
            "PinHits": 968,
            "Pins": 969,
            "Gets": 969,
            "GetHits": 968
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 0,
            "Invalidations": 0,
            "NameSpace": "PDBOPER",
            "PinHits": 0,
            "Pins": 0,
            "Gets": 6,
            "GetHits": 0
          },
       
       
          {
            "InstanceName": "ORCL",
            "Reloads": 4200,
            "Invalidations": 0,
            "NameSpace": "TABLE/PROCEDURE",
            "PinHits": 294879,
            "Pins": 318917,
            "Gets": 136422,
            "GetHits": 124010
          },
          {
            "InstanceName": "ORCL",
            "Reloads": 4031,
            "Invalidations": 3298,
            "NameSpace": "SQL AREA",
            "PinHits": 3813428,
            "Pins": 3850249,
            "Gets": 161629,
            "GetHits": 146974
          }
        ],
        "BufferPoolStatistics": [
          {
            "InstanceName": "ORCL",
            "FreeBufferWait": 0,
            "PhysicalReads": 78823,
            "DbBlockGets": 4243203,
            "DbBlockChange": 4531798,
            "ConsistentGets": 231688190,
            "BufferBusyWait": 428,
            "Name": "DEFAULT",
            "PhysicalWrites": 162190
          }
        ],
        "SysTimeModel": [
          {
            "InstanceName": "ORCL",
            "Value": 938,
            "StatName": "DB CPU"
          },
          {
            "InstanceName": "ORCL",
            "Value": 867712,
            "StatName": "DB time"
          }
        ]
      }
    ],
    "hostname": "DESKTOP-GK9AKCD",
    "@timestamp": "2021-04-05T10:11:36+05:30",
    "client.geo.location": null,
    "ServiceName": "performance",
    "ServiceStatus": "0",
    "labels.Country": null,
    "labels.Age": "",
    "ServiceSummary": "OK - Buffer hit ratio: 99.97%, Library cache hit ratio: 98.13%",
    "labels.AlertFlag": "No",
    "labels.OrgId": null,
    "labels.RegisterNum": null
  },
  "fields": {
    "@timestamp": [
      "2021-04-05T04:41:36.000Z"
    ],
    "Memory Usage": [
      0
    ],
    "CPU Status": [
      ""
    ],
    "MemoryUsageStatus": [
      ""
    ],
    "EventTypeTime": [
      null
    ]
  },
  "sort": [
    1617597696000
  ]
}

```

Above is a sample of various kinds of documents in my index.  
There is no labels.region in this doc.You can my be pick labels.Country instead of that

Pls use labels.OrgId, labels.StoreNum, labels.RegisterNum  
(these names have a dot operator in it. So I have a challenge in accessing them as well) from the doc.

Pls dont use the ones inside OrgHier.

In the third document certain values are empty /null and thats the expected behaviour. So we need to have null checks for that . pls use the above 3 documents for your testing.  
Thanyou @markov00

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 5, 2021, 10:24am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/15 "2021-04-05T10:24:36Z")

</div>

My current Vega Spec

```auto
{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": "container",
  "height": "container",
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "format": {
        "property": "hits.hits"
      },
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum['labels.OrgId']"
        },
        {
          "type": "fold",
          "fields": [
            "_source.OrgId",
            "_source.Region",
            "_source.StoreNum",
            "_source.RegisterNum"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.OrgId' ? datum._source.OrgId : datum.key == '_source.Region' ? join([datum._source.labels.OrgId,datum._source.region],'-') : datum.key == '_source.StoreNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum, datum._source.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == '_source.labels.OrgId' ? null : datum.key == '_source.Region' ? datum._source.labels.OrgId : datum.key == '_source.StoreNum' ? join([datum._source.labels.OrgId, datum._source.Region],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.labels.OrgId, datum._source.Region, datum._source.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ],
      "values": {
        "took": 1,
        "timed_out": false,
        "_shards": {
          "total": 23,
          "successful": 23,
          "skipped": 21,
          "failed": 0
        },
        "hits": {
          "total": 28,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "GzmNoXgBQh5uzqdXvqs2",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T10:22:04.329946300Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Payment_Device",
                "ServiceSummary": "PDQ Device is online",
                "ServiceStatus": "0",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:23:22.231Z",
                "labels.Age": "178",
                "ServiceDetail": [
                  {
                    "Status": "ONLINE",
                    "PaymentDeviceIP": "128.100.5.29"
                  }
                ]
              }
            }
          ]
        },
        "aggregations": {
          "time_buckets": {
            "buckets": []
          }
        }
      }
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

```

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [April 5, 2021, 12:42pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/16 "2021-04-05T12:42:39Z")

</div>

Not looking at the whole picture here but started going down the list off your transformations and the first one doesn't return a result.

Try writing it this to get a value for `OrgId`

`{"type": "formula", "as": "OrgId", "expr": "datum._source['labels.OrgId']"}`

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 5, 2021, 2:31pm UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/17 "2021-04-05T14:31:47Z")

</div>

> [@aaron-nimocks](#):
>
> {"type": "formula", "as": "OrgId", "expr": "datum.\_source['labels.OrgId']"}

Thanks @aaron-nimocks .

I have the error Infinite extent for field "depth": [Infinity, -Infinity] now ...Pls assist.

I have tried that and here is my current spec

```auto
{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": "container",
  "height": "container",
  "padding": 5,
  "signals": [
    {
      "name": "labels",
      "value": true,
      "bind": {
        "input": "checkbox"
      }
    },
    {
      "name": "layout",
      "value": "tidy",
      "bind": {
        "input": "radio",
        "options": [
          "tidy",
          "cluster"
        ]
      }
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": [
          "line",
          "curve",
          "diagonal",
          "orthogonal"
        ]
      }
    },
    {
      "name": "separation",
      "value": false,
      "bind": {
        "input": "checkbox"
      }
    }
  ],
  "data": [
    {
      "name": "table",
      "format": {
        "property": "hits.hits"
      },
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum._source['labels.OrgId']"
        },
        {
          "type": "formula",
          "as": "Region",
          "expr": "datum._source['labels.Region']"
        },
        {
          "type": "formula",
          "as": "StoreNum",
          "expr": "datum._source['labels.StoreNum']"
        },
        {
          "type": "formula",
          "as": "RegisterNum",
          "expr": "datum._source['labels.RegisterNum']"
        },
        {
          "type": "fold",
          "fields": [
            "_source.OrgId",
            "_source.Region",
            "_source.StoreNum",
            "_source.RegisterNum"
          ]
        },
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == '_source.OrgId' ? datum._source.OrgId : datum.key == '_source.Region' ? join([datum._source.OrgId,datum._source.Region],'-') : datum.key == '_source.StoreNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum, datum._source.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == '_source.OrgId' ? null : datum.key == '_source.Region' ? datum._source.OrgId : datum.key == '_source.StoreNum' ? join([datum._source.OrgId, datum._source.Region],'-') : datum.key == '_source.RegisterNum' ? join([datum._source.OrgId, datum._source.Region, datum._source.StoreNum],'-') : null"
        },
        {
          "type": "filter",
          "expr": "isDefined(datum.value)"
        },
        {
          "type": "aggregate",
          "groupby": [
            "id",
            "parent"
          ]
        },
        {
          "type": "formula",
          "as": "label",
          "expr": "peek(split(datum.id,'-'))"
        }
      ],
      "values": {
        "took": 11,
        "timed_out": false,
        "_shards": {
          "total": 23,
          "successful": 23,
          "skipped": 21,
          "failed": 0
        },
        "hits": {
          "total": 2706,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "FyUIoXgBJTwroTEjoWCu",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T07:56:40.636197100Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Application",
                "ServiceSummary": "Xstore Application is down ",
                "ServiceStatus": "2",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:19:31.034Z",
                "labels.Age": "37",
                "ServiceDetail": [
                  {
                    "Xstore": "OFFLINE",
                    "JmxPort": "2020",
                    "ConnectionToPort": "OFFLINE"
                  }
                ]
              },
              "OrgId": "1000",
              "StoreNum": "101",
              "RegisterNum": "2"
            }
          ]
        },
        "aggregations": {
          "time_buckets": {
            "buckets": []
          }
        }
      }
    },
    {
      "name": "tree",
      "source": "table",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "tree",
          "method": {
            "signal": "layout"
          },
          "size": [
            {
              "signal": "height"
            },
            {
              "signal": "width - 100"
            }
          ],
          "separation": {
            "signal": "separation"
          },
          "as": [
            "y",
            "x",
            "depth",
            "children"
          ]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {
          "type": "treelinks"
        },
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {
            "signal": "links"
          }
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {
        "scheme": "magma"
      },
      "domain": {
        "data": "tree",
        "field": "depth"
      },
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {
        "data": "links"
      },
      "encode": {
        "update": {
          "path": {
            "field": "path"
          },
          "stroke": {
            "value": "#ccc"
          }
        }
      }
    },
    {
      "type": "symbol",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "size": {
            "value": 100
          },
          "stroke": {
            "value": "#fff"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "fill": {
            "scale": "color",
            "field": "depth"
          }
        }
      }
    },
    {
      "type": "text",
      "from": {
        "data": "tree"
      },
      "encode": {
        "enter": {
          "text": {
            "field": "label"
          },
          "fontSize": {
            "value": 9
          },
          "baseline": {
            "value": "middle"
          }
        },
        "update": {
          "x": {
            "field": "x"
          },
          "y": {
            "field": "y"
          },
          "dx": {
            "signal": "datum.children ? -7 : 7"
          },
          "align": {
            "signal": "datum.children ? 'right' : 'left'"
          },
          "opacity": {
            "signal": "labels ? 1 : 0"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

```

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 6, 2021, 9:36am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/18 "2021-04-06T09:36:22Z")

</div>

There is a bit of confusion about what is the `_source` `datum` and the created fields with the `formula` transformation.  
I've cleaned up it a bit and now it looks fine at least on the vega editor online.  
I've also slightly changed the way you get the `Region` as it looks like not always available in the labels object but can be part of the `OrgHier` object.

```auto
{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "description": "An example of Cartesian layouts for a node-link diagram of hierarchical data.",
  "width": 1000,
  "height": 800,
  "padding": 5,
  "signals": [
    {"name": "labels", "value": true, "bind": {"input": "checkbox"}},
    {
      "name": "layout",
      "value": "tidy",
      "bind": {"input": "radio", "options": ["tidy", "cluster"]}
    },
    {
      "name": "links",
      "value": "diagonal",
      "bind": {
        "input": "select",
        "options": ["line", "curve", "diagonal", "orthogonal"]
      }
    },
    {"name": "separation", "value": false, "bind": {"input": "checkbox"}}
  ],
  "data": [
    {
      "name": "table",
      "format": {"property": "hits.hits"},
      "transform": [
        {
          "type": "formula",
          "as": "OrgId",
          "expr": "datum._source['labels.OrgId']"
        },
        {
          "type": "formula",
          "as": "Region",
          "expr": "datum._source['labels.Region'] || datum._source.OrgHier.Region"
        },
        {
          "type": "formula",
          "as": "StoreNum",
          "expr": "datum._source['labels.StoreNum']"
        },
        {
          "type": "formula",
          "as": "RegisterNum",
          "expr": "datum._source['labels.RegisterNum']"
        },
        {
          "type": "fold",
          "fields": ["OrgId", "Region", "StoreNum", "RegisterNum"]
        }
      ],
      "values": {
        "took": 11,
        "timed_out": false,
        "_shards": {"total": 23, "successful": 23, "skipped": 21, "failed": 0},
        "hits": {
          "total": 2706,
          "max_score": 1,
          "hits": [
            {
              "_index": "apm-retmon",
              "_type": "_doc",
              "_id": "FyUIoXgBJTwroTEjoWCu",
              "_score": 1,
              "_source": {
                "@timestamp": "2021-04-05T07:56:40.636197100Z",
                "hostname": "SIMPHONY3",
                "client.geo.location": "41.378197,-81.462354",
                "ServiceName": "POS_Application",
                "ServiceSummary": "Xstore Application is down ",
                "ServiceStatus": "2",
                "labels.OrgId": "1000",
                "labels.Country": "US",
                "labels.StoreNum": "101",
                "labels.LatLong": "41.378197,-81.462354",
                "labels.RegisterNum": "2",
                "labels.OpType": "Availability",
                "OrgHier": {
                  "OrgId": 1000,
                  "RegsterNum": "2",
                  "Store": "101",
                  "District": "1",
                  "Region": "West"
                },
                "labels.AppType": "xstore",
                "labels.AlertFlag": "No",
                "StatusChangeTime": "2021-04-05T07:19:31.034Z",
                "labels.Age": "37",
                "ServiceDetail": [
                  {
                    "Xstore": "OFFLINE",
                    "JmxPort": "2020",
                    "ConnectionToPort": "OFFLINE"
                  }
                ]
              },
              "OrgId": "1000",
              "StoreNum": "101",
              "RegisterNum": "2",
              "Region": "West"
            }
          ]
        },
        "aggregations": {"time_buckets": {"buckets": []}}
      }
    },
    {
      "source": "table",
      "name": "transformation",
      "transform": [
        {
          "type": "formula",
          "as": "id",
          "expr": "datum.key == 'OrgId' ? datum.OrgId : datum.key == 'Region' ? join([datum.OrgId,datum.Region],'-') : datum.key == 'StoreNum' ? join([datum.OrgId, datum.Region, datum.StoreNum],'-') : datum.key == 'RegisterNum' ? join([datum.OrgId, datum.Region, datum.StoreNum, datum.RegisterNum],'-') : null"
        },
        {
          "type": "formula",
          "as": "parent",
          "expr": "datum.key == 'OrgId' ? null : datum.key == 'Region' ? datum.OrgId : datum.key == 'StoreNum' ? join([datum.OrgId, datum.Region],'-') : datum.key == 'RegisterNum' ? join([datum.OrgId, datum.Region, datum.StoreNum],'-') : null"
        },
        {"type": "filter", "expr": "isDefined(datum.value)"},
        {"type": "aggregate", "groupby": ["id", "parent"]},
        {"type": "formula", "as": "label", "expr": "peek(split(datum.id,'-'))"}
      ]
    },
    {
      "name": "tree",
      "source": "transformation",
      "transform": [
        {"type": "stratify", "key": "id", "parentKey": "parent"},
        {
          "type": "tree",
          "method": {"signal": "layout"},
          "size": [{"signal": "height"}, {"signal": "width - 100"}],
          "separation": {"signal": "separation"},
          "as": ["y", "x", "depth", "children"]
        }
      ]
    },
    {
      "name": "links",
      "source": "tree",
      "transform": [
        {"type": "treelinks"},
        {
          "type": "linkpath",
          "orient": "horizontal",
          "shape": {"signal": "links"}
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "color",
      "type": "linear",
      "range": {"scheme": "magma"},
      "domain": {"data": "tree", "field": "depth"},
      "zero": true
    }
  ],
  "marks": [
    {
      "type": "path",
      "from": {"data": "links"},
      "encode": {
        "update": {"path": {"field": "path"}, "stroke": {"value": "#ccc"}}
      }
    },
    {
      "type": "symbol",
      "from": {"data": "tree"},
      "encode": {
        "enter": {"size": {"value": 100}, "stroke": {"value": "#fff"}},
        "update": {
          "x": {"field": "x"},
          "y": {"field": "y"},
          "fill": {"scale": "color", "field": "depth"}
        }
      }
    },
    {
      "type": "text",
      "from": {"data": "tree"},
      "encode": {
        "enter": {
          "text": {"field": "label"},
          "fontSize": {"value": 9},
          "baseline": {"value": "middle"}
        },
        "update": {
          "x": {"field": "x"},
          "y": {"field": "y"},
          "dx": {"signal": "datum.children ? -7 : 7"},
          "align": {"signal": "datum.children ? 'right' : 'left'"},
          "opacity": {"signal": "labels ? 1 : 0"}
        }
      }
    }
  ],
  "config": {
    "range": {"category": {"scheme": "elastic"}},
    "arc": {"fill": "#54B399"},
    "area": {"fill": "#54B399"},
    "line": {"stroke": "#54B399"},
    "path": {"stroke": "#54B399"},
    "rect": {"fill": "#54B399"},
    "rule": {"stroke": "#54B399"},
    "shape": {"stroke": "#54B399"},
    "symbol": {"fill": "#54B399"},
    "trail": {"fill": "#54B399"},
    "title": {"color": "#343741"},
    "style": {
      "guide-label": {"fill": "#69707d"},
      "guide-title": {"fill": "#343741"},
      "group-title": {"fill": "#343741"},
      "group-subtitle": {"fill": "#343741"}
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "autosize": {"type": "fit", "contains": "padding"}
}

```

---

<div class="post-metadata">

**Author:** ![anjana1](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@anjana1](https://discuss.elastic.co/u/anjana1)\
**Post date:** [April 6, 2021, 9:40am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/19 "2021-04-06T09:40:54Z")

</div>

Thanks.. Regarding the Region .. suppose if we dont have Region at all in a document how can we make it skip that document .. I was able to create the tree flow dynamically . Thanks to the team . Now my question is, if any of the fields are null/empty the script should not fail, that document need to be skipped.

---

<div class="post-metadata">

**Author:** ![markov00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markov00/32/33316_2.png) [@markov00](https://discuss.elastic.co/u/markov00)\
**Post date:** [April 6, 2021, 10:12am UTC](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993/20 "2021-04-06T10:12:11Z")

</div>

Does the document need to be skipped also if just one field is empty/null?  
If so you can also use ES for that, adding an `exists` query to your existing query [Exists query | Elasticsearch Guide [7.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html)

[Next page](https://discuss.elastic.co/t/dynamic-fetch-data-from-index-to-generate-treeflow/268993.md?page=2)