Real-time visualization of numbers in arrays from json files

[Dimitris_S]

Hi there, total newcomer in the ElasticSearch concept. I've been reading the manuals for quite some time but I have managed minimal progress. Here is my scenario:

1. From a Raspberry Pi I collect some sensor data using Python and I group them in a JSON format. The complete JSON message is then sent via UDP to the ElasticSearch via FileBeat. This step is successful and with the option [tags] in filebeat.yml I'm able to filter easily my data in Kibana.

2. The JSON messages are a bit complex. They contain fields for timestamps, device ID's, message counters and a Payload list. This list includes then several measurements taken in a predefined time frame. Each measurement includes 2 arrays for storing HW statistics and analog measurements for the current. Below is an example with just 2 measurements:
   {
   "timestamp":"09-12-2019 14:06:10",
   "device_id":"C1333",
   "msg_counter":"000",
   "payload":[
   {
   "cc":"01",
   "timestamp":"09-12-2019 14:06:09",
   "hw_stats":[
   51.5,
   2.4,
   163.9
   ],
   "current":[
   87,
   11.86,
   6.66
   ]
   },
   {
   "cc":"02",
   "timestamp":"09-12-2019 14:06:09",
   "hw_stats":[
   51.5,
   2.4,
   163.7
   ],
   "current":[
   68,
   12.20,
   6.69
   ]
   }
   ]
   }

3. My goal is to visualize each of the values in the "hw_stats" and "current" arrays real-time as the Rasberry uploads UDP data.

4. From the reading I've done I realize that I probably need to use nested data but still I'm not able to parse individual values.

5. Can someone suggest a work plan? I.e., i) is my JSON file formatted correctly? ii) do I need to change mappings to declare that "current" and "hw_stats" are arrays? iii) how do I really access individual data and visualize them? For the time being I just want to display the values in a nice graphical environment real-time but later on I plan to do analysis on them.

[Marius_Dragomir]

I'd highly recommend to use just simple fields for each part of the nested object since nested fields are not fully supported in Kibana so you would have limitations doing various stuff with your data in Kibana. And for displaying the data as it comes, look at using Visual Builder(TSVB) as it displays data in a timeseries, based on your selected timestamp.