How to get arrays with variable length into Elasticsearch?

Elastic Stack Elasticsearch
1

I have log files with a JSON data structure that looks like this:

{"windowNumber": 1830323, "timestamp": "2020-09-24T01:26:31.673097Z", "tagData": {"103040840050DDABCD10028300003B93": {"inIntensity": "2542", "inCount": 2}, "103040840004FCA60A64028108F4C764": {"outIntensity": "760", "outCount": 2}, "10304084004CCB348D100283000023D8": {"inIntensity": "90", "inCount": 1}, "10304084005190348D100283000023B7": {"inIntensity": "51", "inCount": 1}, "103040840006FCD0D36E028108F339A0": {"inIntensity": "6848", "inCount": 5, "outIntensity": "2246", "outCount": 2}, "08304DDD0BD2B97F29D551AF40620400": {"inIntensity": "2513", "inCount": 4}, "103040840006FCA5E86E828108F33E35": {"inIntensity": "2437", "inCount": 3}}, "cycleCount": 6, "reader": "xspan-13-e4-42"}

{"windowNumber": 1830324, "timestamp": "2020-09-24T01:26:31.975344Z", "tagData": {"103040840008C8A581E7828400003788": {"inIntensity": "25", "inCount": 1}, "103040840050DDABCD10028300003B93": {"inIntensity": "6485", "inCount": 4, "outIntensity": "802", "outCount": 1}, "103040840004FCA60A64028108F4C764": {"outIntensity": "254", "outCount": 1}, "10304084005CDDABCD10028300003B96": {"inIntensity": "1426", "inCount": 1}, "10304084005190348D100283000023B7": {"inIntensity": "172", "inCount": 3}, "103040840006FCD0D36E028108F339A0": {"inIntensity": "8162", "inCount": 5}, "08304DDD0BD2B97F29D551AF40620400": {"inIntensity": "3741", "inCount": 3, "outIntensity": "568", "outCount": 1}, "10304084005190348D100283000023CE": {"outIntensity": "45", "outCount": 1}, "103040840006FCA5E86E828108F33E35": {"inIntensity": "3531", "inCount": 2}}, "cycleCount": 7, "reader": "xspan-13-e4-42"}

{"windowNumber": 1830325, "timestamp": "2020-09-24T01:26:32.277731Z", "tagData": {"103040840008C8A581E7828400003788": {"inIntensity": "16", "inCount": 1}, "103040840050DDABCD10028300003B93": {"outIntensity": "2145", "outCount": 3}, "103040840004FCA60A64028108F4C764": {"outIntensity": "285", "outCount": 1}, "1030408400C3A2A58DB6028108F33B23": {"inIntensity": "113", "inCount": 1}, "10304084004CCB348D100283000023D8": {"inIntensity": "113", "inCount": 1, "outIntensity": "68", "outCount": 2}, "103040840006FCD0D36E028108F339A0": {"inIntensity": "6069", "inCount": 3, "outIntensity": "1600", "outCount": 1}, "10304084005190348D100283000023B7": {"outIntensity": "45", "outCount": 1}, "08304DDD0BD2B97F29D551AF40620400": {"inIntensity": "7452", "inCount": 6}, "103040840006FCA5E86E828108F33E35": {"inIntensity": "2816", "inCount": 2}}, "cycleCount": 6, "reader": "xspan-13-e4-42"}

As you can see, tagData is an array with variable length and every tagData object has its own property values and not every tagData object has all the properties.

How would you pre-process this data so it can be visualised in Kibana based on the properties for each tagData object?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.