Automation of adding parent to child relation fields into kibana visualization

[Ahmedeyhaab (Ahmed Ehab Mohamed)]

Hi,

I have a use case that I upload json file directly to elasticsearch with the following data structure

{
  "field_1": "string_value_1",
  "field_2": "numerical_value_2",
  "${parent_field}": {
    "${parent_field}_${child_field_1}": "child_numerical_value_1",
    "${parent_field}_${child_field_2}": "child_numerical_value_2",
    "${parent_field}_${child_field_3}": "child_numerical_value_3",
    "..":".."
  }
}

the index mapping something looks like below:-

{
  "mappings": {
    "_doc": {
      "properties": {
        "field_1": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "field_2": {
          "type": "long"
        },
        "${parent_field}": {
          "properties": {
            "${parent_field}_${child_field_1}": {
              "type": "long"
            },
            "${parent_field}_${child_field_2}": {
              "type": "long"
            },
            "${parent_field}_${child_field_3}": {
              "type": "long"
            }
          }
        }
      }
    }
  }
}

Whenever I tried to visualize the child fields into kibana line graph for example, I manually add all the child fields one by one, however I need something to automate the adding of child fields in the visualization without manually adding especially when there is a new child field is added later, kibana could automatically detect the new child field and add it to the visualization.

Is there something feasible for use case and how could this be implemented?

I am using Elasticsearch and kibana version 7.12.1 and building them using docker containers.

[lukas (Lukas Olson)]

It honestly sounds like you should probably be using nested fields, which aren't well supported in Kibana. See Nested field support · Issue #1084 · elastic/kibana · GitHub.

It's also not totally clear how you would like to visualize this data. If you could give me an example of what you'd like to accomplish, I might be able to suggest something using runtime fields.

[Ahmedeyhaab (Ahmed Ehab Mohamed)]

Hi,
Thanks for the quick answers.

All I need is to have a line graph for all the child member field from parent_field, something as below:-

if I have a lot of child fields (for example 40 child fields), I add it manually and the maintenance for these child fields are to hard and I need something to dynamically visualize all the child fields without any manual intervention and auto detection if there is a new child field has been added to the parent later in the future.

I hope this gets more clear now my use case.

Thanks

[lukas (Lukas Olson)]

Hmm, unfortunately I can't think of something quickly to suit your use case.

Is it possible for you to change the format of the data being indexed into Elasticsearch? It would be much easier to accomplish if you had separate documents for each of your child fields. For example, instead of just one document that looks like this:

{
    "field_1": "string_value_1",
    "field_2": "numerical_value_2",
    "${parent_field}": {
        "${parent_field}_${child_field_1}": "child_numerical_value_1",
        "${parent_field}_${child_field_2}": "child_numerical_value_2",
        "${parent_field}_${child_field_3}": "child_numerical_value_3"
    }
}

You could have the following three documents:

{
    "field_1": "string_value_1",
    "field_2": "numerical_value_2",
    "child_field_type": "${parent_field}_${child_field_1}",
    "child_field_value": "child_numerical_value_1"
}

{
    "field_1": "string_value_1",
    "field_2": "numerical_value_2",
    "child_field_type": "${parent_field}_${child_field_2}",
    "child_field_value": "child_numerical_value_2"
}

{
    "field_1": "string_value_1",
    "field_2": "numerical_value_2",
    "child_field_type": "${parent_field}_${child_field_3}",
    "child_field_value": "child_numerical_value_3"
}

And then you could create a visualization, splitting terms by child_field_type.

[Ahmedeyhaab (Ahmed Ehab Mohamed)]

Hi @lukas,

I got your workaround, but unfortunately, it's hard to restructure all the fields again in separate documents like the way you have illustrated above in your last comment.

Thanks