Separate Fields Now Displaying As One Object

Elastic Stack Kibana
1

I use Logstash to process our logs and organize relevant fields into objects so that they display together in Kibana.

log.source
log.format
log.time

However, after upgrading, I THINK to 7.16, they're now displaying as an object under the root field log in JSON.

{
  "format": "",
  "source": "",
  "time": ""
}

When did this happen? Is there a setting I can change or do I have to completely re-write my Logstash pipelines to put them in separate fields?

2

Could you share the mapping of your index?

3 
            "log": {
                "properties": {
                    "format": {
                        "type": "keyword"
                    },
                    "time": {
                        "type": "date"
                    },
                    "source": {
                        "type": "keyword"
                    }
                }
            },
4

This is should work fine, can you share a screenshot of the problem?

5

The fields in my previous example aren't real data, but below is the same problem with real data.

            "action": {
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            },

Kibana Screenshot

image

6

What happen when you reload the kibana page?
The mappings are the same across all the indices contained in the index pattern?

7

Same behavior reloading the page.

I've deleted all the related indices to eliminate the possibility of data type mismatch.

8

For Elasticsearch, this;

log.source
log.format
log.time

Is the same as this;

{
   "log":
   {
     "format": "",
     "source": "",
     "time": ""
   }
}

So Kibana is just displaying them differently, but to Elasticsearch there's no change.

9

Are you sure all indices in your index pattern have this mapping? I'm guessing there is a conflict and some index has another mapping. I tested the following on 7.16:

PUT actionindex
{
  "mappings": {
    "properties": {
     "action": {
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            }
  } 
    }
}

POST actionindex/_doc
{
  "action": {
    "detail": "a test",
    "endpoint": "the_endpoint",
    "method": "the_method"
  }
}

This is how it looks like in discover (all fields are recognized correctly):

Screenshot 2022-01-12 at 09.48.17
Screenshot 2022-01-12 at 09.48.171684×800 67.2 KB

10

I stopped Filebeat on the remote server that pushes the logs, deleted all indices, the index template, the index pattern from Kibana/Elasticsearch, and restarted the pipeline to push a fresh copy of the template into Elasticsearch. Still getting the same appearance in Kibana.

JSON snippet of event

{
  "_source": {
    "timestamp": "2021-12-30t11:11:39.722296",
    "action": {
      "detail": "",
      "endpoint": "/validate/check",
      "method": "post"
    },

Table view of event
image

Index Template Configuration

            "action": {
				"type": "object",
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            },
11

I just discovered the cause of the issue, which may be a regression bug. When the option "Discover fields from source" is enabled, it causes the unwanted behavior. Once I disabled this setting, the fields started displaying properly. However, I enabled this option in the past because not all fields were being displayed in my indices in the past. Hoping that problem has been fixed...

12

Just wanted to add that I've reproduced and opened an issue for this case: [Discover] Using _source fields of objects are no longer displayed separately · Issue #123169 · elastic/kibana · GitHub

thx a lot for reporting!