Mapping dynamic JSON

[FrankyFire]

Hello there,

I have an Inputfile that looks as follows:

"message": {
    "Name1": {
        "color": "blue",
        "information": {
            "origin": "East"
        }
    },
    "Name2" {...}
}

Where "Name2" has the same hierarchy as "Name1". Also, there can be more or less of every attribute in "message" and the order can change.
I am able to map "message" as a nested field, but can't do so with it's children, as I don't know the names of them in the first place. Using "wildcard" does'nt seem to work in a PUT argument.

The goal is accessing the fields values like "Name1.color" and "Name2.information.origin", but without losing context of the parent it's inherited by. In Kibana I can search for the attributes, but not for the parent. I would like to have it grouped by the parent name, to see every of its attributes at once.

How do I achieve this?

I already tried using the KV-Filter Plugin in Logstash, that returns each Value and it's parent. But that loses the context to the parent "NameX" when applied to the attributes in "information".

[FrankyFire]

I was able to partly solve the issue. There is a pattern for what I want to do. I simply rename every Object in the second hierarchy level. Sadly this only works for me, as the name occurs again in the third hierarchy level.
To do this, you first have to change the codec of the input file to plain, as the

mutate { gsub=> [ ] } 

does'nt seem to work with JSON files. Afterwards check what the file looks like that you will get returned. It slightly changed for me. Afterwards you can match everything in the message field for the pattern you use and exclude the fields you don't want to change (for me this is "information", as it has the same pattern but is already static). This is what did the trick for me:

mutate {
    gsub => [ "message","\"[^(\"message|\"information)][^\"]+\": {","\"Name\": {" ]
}

What works great, but somehow leaves me with tripled blackslashes. So you may need to clean this up. I used the same gsub sequence for this and added a double backslash like this:

"message","[\\]{1}",""

This seem to have a bug as well, therefor it has to look like this. Afterwards I added a JSON-Filter, which works great! ....for the first Element. It breaks after the }, sign. This leaves me with my next problem: There is no recursive Option. How do I parse the all Elements in the message field?

Edit: Changed topic to Logstash, as I'm moving away from an Elasticsearch solution.

Edit 2: I wanted to share this Website, as this helps testing your regex quickly.