Index mapping shows key as "object", Kibana Management as "string"

[A_B]

Hello all,

I'm trying to understand why Kibana > Management > Index Patterns shows me one thing and using Dev Tools to query API shows me another.

Some of our logs have the key json.user_agent.

In Kibana > Management > Index Patterns I see the following (I have tried refreshing many times)

In the console when I look at the key it says

GET _mapping/field/json.user_agent
{
  "index-name" : {
    "mappings" : { }
  },
....
}

or alternatively

GET /index-name/_mapping
            "user_agent" : {
              "type" : "object"
            },

So, looks like the true type is object but Kibana thinks it is string for some reason.

There is no template that sets the type for this key.

ES and Kibana version 7.8.1, both upgraded recently from older versions.

The same is true in version 7.8.0 as well in environments where I have not yet upgraded to the latest version.

Any tips on how to reconcile this?

Cheers,
AB

[mattkime (Matthew Kime)]

Hello @A_B

Are you able to share the full results of these queries? I suspect there might be confusion over field names but I can't determine that from partial information.

Thanks,
Matt

[A_B]

Hi @mattkime,

thank you for the reply

We have historically had significant problems with data type collisions for keys because we do not have any internal common schema for logs

Usually Kibana would notice this and report the field as having a conflict between indices matching the index pattern.

We use fairly broad index patterns in Kibana. My following example will show 6 matches for index_name_prefix per date

$ for i in $(curl -s http://elasticsearch:9200/_cat/indices | grep index_name_prefix | grep 2020.07.27 | cut -d ' ' -f 3); do curl -s http://elasticsearch:9200/$i/_mapping | jq '.["'${i}'"].mappings.properties.json.properties.user_agent' ; done
null
null
null
null
{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}
null
$ for i in $(curl -s http://elasticsearch:9200/_cat/indices | grep index_name_prefix | grep 2020.08.05 | cut -d ' ' -f 3); do curl -s http://elasticsearch:9200/$i/_mapping | jq '.["'${i}'"].mappings.properties.json.properties.user_agent' ; done
null
null
null
null
{
  "type": "object"
}
null

As you can see, the data type is not the same in those two instances where this key exists. So I guess, what was unexpected was that Kibana did not report this key as having data type conflicts.

As I suspected, the root cause will be our poor log schema, I was just confused by the reporting in Kibana.

[mattkime (Matthew Kime)]

I'm don't yet have a specific answer and agree that the reporting is a bit confusing. That said, it sounds like the schema is likely the problem.

Have you looked at ECS? https://www.elastic.co/guide/en/ecs/current/ecs-reference.html

[A_B]

I did not submit the full mapping as it have a lot of keys (1000+). The mapping in question is not ECS compatible. json.user_agent is a pre-ECS convention we have used. We are working on adapting ECS which uses user_agent.*

That still does not change the fact that Kibana does not tell me that there is a conflict in data type for the specific key between the indices matching the index pattern I have configured, which is what confused me while troubleshooting some issues.

This thread can be closed