Json fields not placed in elasticsearch properly

[Gauti (Gautham)]

HI All,

We have ingested a json file of data through logstash, now the data has not been properly placed in elasticsearch without this we cant utilize this data to build dashboards in kibana.

JSON file for your reference.

{
  "_index": "perflog2",
  "_type": "_doc",
  "_id": "j_eJuBtAgfD7CrH",
  "_version": 1,
  "_score": null,
  "_source": {
    "MEM": [
      {
        "data": 76.56,
        "collection_time": 1566442800
      },
      {
        "data": 76.01,
        "collection_time": 1566443100
      },
      {
        "data": 75.95,
        "collection_time": 1566443400
      }
    ],
    "@timestamp": "2019-08-22T13:35:12.538Z",
    "SWAP": [
      {
        "data": 67.86,
        "collection_time": 1566442800
      },
      {
        "data": 67.6,
        "collection_time": 1566443100
      },
      {
        "data": 67.69,
        "collection_time": 1566443400
      }
    ],
    "path": "/opt/curl_output/output.json",
    "CPU": [
      {
        "data": 60.55,
        "collection_time": 1566442800
      },
      {
        "data": 35.94,
        "collection_time": 1566443100
      },
      {
        "data": 39.46,
        "collection_time": 1566443400
      }
    ],
    "hostname": "1.1.1.6",
    "@version": "1",
    "host": "Logstash",
    "Filesystem": [
      [
        {
          "d_used_percent": "64.9",
          "collection_time": "1566442800",
          "index_label": "C:\\"
        }
      ],
      [
        {
          "d_used_percent": "0.6",
          "collection_time": "1566442800",
          "index_label": "F:\\"
        }
      ],
      [
        {
          "d_used_percent": "27.6",
          "collection_time": "1566442800",
          "index_label": "D:\\"
        }
      ]
    ]
  },
  "fields": {
    "@timestamp": [
      "2019-08-22T13:35:12.538Z"
    ]
  },
  "highlight": {
    "hostname.keyword": [
      "@kibana-highlighted-field@1.1.1.6@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1566480912538
  ]
}

Basically its the performance data of a host. Can someone help me out on how to format it and make it a meaning full data for kibana visualization.

Thanks
Gauti

[spinscale (Alexander Reelsen)]

from a quick look it seems to me you have combined three events into one, despite those three events happened at different times. I think it makes sense to unroll those three events, so that you are storing three documents. This way you also will not need to store arrays for each category.

[Gauti (Gautham)]

@spinscale but how do i unroll those three events, its basically coming from one of my monitoring tool in this format, how can i convert as a useful data in elasticsearch.
Atleast for CPU,RAM and SWAP data all are integers so was able to do a sum of data and showcase in kibana, but when it comes to FILESYSTEM data, there is a string value(Drive Letter) due to which i'm not able to showcase data for filesystem.

Any advice on this please...

Thanks
Gautham