Separate Fields Now Displaying As One Object

[wwalker (Walker)]

I use Logstash to process our logs and organize relevant fields into objects so that they display together in Kibana.

log.source
log.format
log.time

However, after upgrading, I THINK to 7.16, they're now displaying as an object under the root field log in JSON.

{
  "format": "",
  "source": "",
  "time": ""
}

When did this happen? Is there a setting I can change or do I have to completely re-write my Logstash pipelines to put them in separate fields?

[flash1293 (Joe Reuter)]

Could you share the mapping of your index?

[wwalker (Walker)]

            "log": {
                "properties": {
                    "format": {
                        "type": "keyword"
                    },
                    "time": {
                        "type": "date"
                    },
                    "source": {
                        "type": "keyword"
                    }
                }
            },

[flash1293 (Joe Reuter)]

This is should work fine, can you share a screenshot of the problem?

[wwalker (Walker)]

The fields in my previous example aren't real data, but below is the same problem with real data.

            "action": {
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            },

Kibana Screenshot

[Tomo_M (Tomohiro Mitani)]

What happen when you reload the kibana page?
The mappings are the same across all the indices contained in the index pattern?

[wwalker (Walker)]

Same behavior reloading the page.

I've deleted all the related indices to eliminate the possibility of data type mismatch.

[warkolm (Mark Walkom)]

For Elasticsearch, this;

log.source
log.format
log.time

Is the same as this;

{
   "log":
   {
     "format": "",
     "source": "",
     "time": ""
   }
}

So Kibana is just displaying them differently, but to Elasticsearch there's no change.

[flash1293 (Joe Reuter)]

@wwalker

 "action": {
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            }

Are you sure all indices in your index pattern have this mapping? I'm guessing there is a conflict and some index has another mapping. I tested the following on 7.16:

PUT actionindex
{
  "mappings": {
    "properties": {
     "action": {
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            }
  } 
    }
}

POST actionindex/_doc
{
  "action": {
    "detail": "a test",
    "endpoint": "the_endpoint",
    "method": "the_method"
  }
}

This is how it looks like in discover (all fields are recognized correctly):

[wwalker (Walker)]

I stopped Filebeat on the remote server that pushes the logs, deleted all indices, the index template, the index pattern from Kibana/Elasticsearch, and restarted the pipeline to push a fresh copy of the template into Elasticsearch. Still getting the same appearance in Kibana.

JSON snippet of event

{
  "_source": {
    "timestamp": "2021-12-30t11:11:39.722296",
    "action": {
      "detail": "",
      "endpoint": "/validate/check",
      "method": "post"
    },

Table view of event

Index Template Configuration

            "action": {
				"type": "object",
                "properties": {
                    "detail": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword"
                            }
                        }
                    },
                    "endpoint": {
                        "type": "keyword"
                    },
                    "method": {
                        "type": "keyword"
                    }
                }
            },

[wwalker (Walker)]

I just discovered the cause of the issue, which may be a regression bug. When the option "Discover fields from source" is enabled, it causes the unwanted behavior. Once I disabled this setting, the fields started displaying properly. However, I enabled this option in the past because not all fields were being displayed in my indices in the past. Hoping that problem has been fixed...

[matw (Matthias Wilhelm)]

Just wanted to add that I've reproduced and opened an issue for this case: [Discover] Using _source fields of objects are no longer displayed separately · Issue #123169 · elastic/kibana · GitHub

thx a lot for reporting!