Nested Array in Elastic Search (Using 6.2.3)

Elastic Stack Kibana
1

Hi All,
I'm finding difficulty in reading the data like below in elastic search. I already converted the objects into arrays before inserting it into elastic search. Below is my mapping in Elastic search I can see the nested data type getting applied but I'm not able to view it properly in Kibana as everything is treated as text.

Any help plz ?

My Data

  "dashboardreport": {
        "@name": "dashboardname",
        "@version": "2323232323",
        "@reportdate": "2018-06-11T18:27:35.051+08:00",
        "@description": "",
        "source": {
          "filters": {
            "@summary": "",
          }
        },
        "reportheader": {
          "reportdetails": {
            "user": "username"
          }
        },
        "data": {
          "tasksmonitorsdashlet": {
            "@name": "monitors",
            "@description": "",
            "taskmonitorrecords": {
              "@structuretype": "tree",
              "taskmonitorrecord": {
                "0": {
                  "@header": "Profile0",
                  "@status": "scheduled",
                  "@lastrunresult": "ok",
                  "@system": "Profile0",
                  "taskmonitorrecord": {
                    "@header": "Profile0",
                    "@status": "scheduled",
                    "@lastrunresult": "ok",
                    "@system": "Profile0",
                    "taskmonitorrecord": {
                      "@header": "Task1 Clean Up Task",
                      "@status": "scheduled",
                      "@lastrunresult": "ok",
                      "@schedule": "Clean Up",
                      "@target": "server",
                      "@system": "local"
                    }
                  }
                },
                "1": {
                  "@header": "Profile1",
                  "@status": "text",
                  "@lastrunresult": "unknown",
                  "@system": "Profile1",
                  "taskmonitorrecord": {
                    "@header": "Import",
                    "@status": "text",
                    "@lastrunresult": "unknown",
                    "@system": "Profile1",
                    "taskmonitorrecord": {
                      "@header": "Profile1",
                      "@status": "text",
                      "@lastrunresult": "unknown",
                      "@schedule": "-",
                      "@target": "server",
                      "@system": "Profile1"
                    }
                  }
                },
                "2": {
                  "@header": "Profile2",
                  "@status": "unknown",
                  "@lastrunresult": "ok",
                  "@system": "Profile2",
                  "taskmonitorrecord": {
                    "0": {
                      "@header": "Monitor2",
                      "@status": "suspended",
                      "@lastrunresult": "unknown",
                      "@system": "Profile2",
                      "taskmonitorrecord": {
                        "0": {
                          "@header": "ExecuteMonitor1",
                          "@status": "suspended",
                          "@lastrunresult": "unknown",
                          "@schedule": "Testing",
                          "@system": "Profile2",
                          "taskmonitorrecord": {
                            "@header": "Servertorun",
                            "@status": "suspended",
                            "@lastrunresult": "unknown",
                            "@schedule": "Testing",
                            "@target": "servertorun",
                            "@system": "Profile2"
                          }
                        },
                        "1": {
                            "@header": "ExecuteMonitor1",
                            "@status": "suspended",
                            "@lastrunresult": "unknown",
                            "@schedule": "Testing",
                            "@system": "Profile2",
                            "taskmonitorrecord": {
                              "@header": "Servertorun",
                              "@status": "suspended",
                              "@lastrunresult": "unknown",
                              "@schedule": "Testing",
                              "@target": "servertorun",
                              "@system": "Profile2"
                            }
                        },

My Mapping

PUT taskmonitor
  {
     "mappings": {
       "doc":{
         "properties": {
           "dashboardreport":{
             "properties": {
               "@reportdate":{"type":"date"},
               "data":{
                 "properties":{
                   "tasksmonitorsdashlet":{
                     "properties":{
                       "taskmonitorrecords":{
                         "properties":{
                           "taskmonitorrecord":{"type":"nested"}
                         }
                       }
                     }
                   }
                 }
               }
               
             }
           }
         }
       }
     }
  }

Discovery tab in Kibana
Discovery_Kibana

How i see it in the visualize tab
Visualize_Tab

Any idea on how the mapping should be to have proper aggregation of the data to visualize it as i cannot use wildcards in field names to get all the headers inside arrays

dashboardreport.data.tasksmonitorsdashlet.taskmonitorrecords.taskmonitorrecord.*.

Thanks in Advance

2

Hi Sakthi,

this is unfortunately currently not possible in Kibana. You are using nested types, which do not yet have support in Kibana. Please follow kibana#1084 for information on nested aggregation support in Kibana.

Cheers,
Tim

3

Hi Tim, Good Day! & Thanks for the response. Any suggestions or workaround on how to feed the data into elastic search like the above to be able to be aggregated.

Note : We already converting all the objects and nested objects to arrays and nested arrays.

Thanks in Advance :slight_smile:

4

As long as you are using type: nested somewhere inside your mapping, Kibana will basically be unable to properly aggregate within those objects, which is your issue here. So if you would want to aggregate over it, you would need to make this regular sub properties (like you did for the rest of the tree), but of course that way you'll actually lose that information nested fields give you: what were the original objects within that array.

So I think there is no real "workaround" instead of giving up on nested objects here, or maybe log each object twice, once in a nested field so you have the support of nested objects where you need it, and also within regular fields, so you can make that kind of aggregation you are looking for.

Cheers,
Tim

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.