Flattening JSON string level with other fields

Elastic Stack Kibana
1

I have set up my kibana and Elastic.
I am getting my logs from Fluentbit.
My main app log, which is JSON, is sent by Fluent with other meta fields like this:

{
    "containerId": "foo",
    "clusterId" : "bar",
    "source" : "stderr",
    "log" : "{ type: error, "msg": "cannot connect to redis" }" // My JSON-FORMATTED LOG GOES HERE 
}

I want to be able to extract fields "type" and "msg" from JSON (which is treated as string in ES) and level those with the rest of fields. This enables me to set filters and indexing properly for them.

How should I approach this?

Things I've tried:
In Data view I tried to add a field log.type (considering dot-notation works) but it doesn't work.
In Index Management, I tried to update field mappings but it doesn't allow me editing the index's fields mapping (I have full privileges).

I am looking to have my list of logs to include "type" and "msg" as distinct columns: searchable and filtertable.

2

I'm not familiar with Fluentbit since it isn't an Elastic product. But you might be able to set up an Ingest pipeline and use the JSON processor to convert the JSON string to an object.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.