Store and visualise key value pairs

[Crystal (Seb Christie)]

Hello,

I have a set of key value pairs returned from an external API that I would like to store in Elastic and be able to visualize in Kibana.

The data is returned from the external API as a JSON object containing multiple key value pairs

    {
        "value" : 
        {
            "en_GB": 181,
            "en_US": 86,
            "es_ES": 20,
            "fr_FR": 10
        }
    }

I would like to be able to store this data in Elastic and visualise it in Kibana in a bar chart for example where the y axis is the respective values and each key is a separate bar.

Currently I am taking this json, calculating each value as a percentage and storing it in a .NET dictionary, then adding this dictionary as a property on a larger object to be stored in Elastic as one document. So the final indexed document looks something like

    {
        "_index": "index-name",
        "_source": {
            "Locale": {
                "en_GB": 60.9,
                "en_US": 28.96,
                "es_ES": 6.9,
                "fr_FR": 3.45
            }
        },
        "fields": {
            "Timestamp": [
                "2021-03-20T11:05:29.309Z"
            ]
        },
        "sort": [
            1616238329309
         ]
    }

This works fine and I can manually create some charts in Kibana by selecting each field. This issue with it is that the keys/fields returned from the API will be different each time and there are many possible values so creating visualisations manually from this isnt really feasible.

Is there a way in Kibana to dynamically select the fields to display in a graph, for example by selecting fields by their location in an object or by the parent object name etc?

Or am I going about this in the wrong way and need to restructure the data before sending it to Elastic? The other idea was to send each key as its own document with the key name and its value and somehow link them through an id or something; but this seemed like a bad solution

Any help is greatly appreciated, thanks.

[flash1293 (Joe Reuter)]

Or am I going about this in the wrong way and need to restructure the data before sending it to Elastic? The other idea was to send each key as its own document with the key name and its value and somehow link them through an id or something; but this seemed like a bad solution

This is what I would suggest - ingest each key/value pair as a separate document, then you can aggregate them (something which is not possible for fields in the same document).

{ key: "en_GB", value: 60.9 }
{ key: "en_US", value: 28.96 }
{ key: "es_ES", value: 6.9 }
{ key: "fr_FR" value: 3.45 }

and so on.

For most aggregations it's probably not even necessary to link them up with an id.

[Crystal (Seb Christie)]

Thanks very much for your response, I ended up going with this solution in the end

Thanks!