Log level highlighting in Discover

I’d like to start this topic to understand how Kibana automatically highlights log_level fields with different background colors.

In my case, when I set up Discover, the field in the index is called log_level. Kibana 8.16.0 initially applied very attractive background colors to this field automatically. However, at some point, this highlighting disappeared, and I haven’t been able to restore it.

Even after reverting everything to my original setup, the highlighting is no longer applied. I’m aware that you can manually set a color format for a field by defining background and foreground colors for different values, but the result doesn’t look as clean or visually appealing as the default formatting that was there initially.

I hope we can find an answer here so others can also benefit from improved log readability in Kibana.

Hi @Matt_Janda and welcome to the community!

We got some feedback that such UI changes are not always relevant to the displayed data set. Because of this the mentioned improvements are now scoped per solution type.

In the more recent versions there is a popover explaining how to enable more context-aware features:

So to get logs related improvements for Discover, choose "Observability" in Space settings:

Hope it helps! Thanks for your feedback!

Thank you for your reply. I did and do have it enabled, hence the confusion on my side. Under observability I can see "Logs"... and other. All of them are selected. Do I have to anything else ?

Okay! Let's find out what is missing.

What is your current version of Kibana?
Did they disappear after an upgrade to a newer Kibana version or you stayed on the same version?
Did they disappear on both Dashboard and Discover pages?

Now, I am running Kibana and Elasticsearch 8.17.4 from docker containers. I have recreated containers and volumes i.e. start everything from scratch. I have created How they disappeared is the mystery part. I have been using 8.16.0, when they disappeared without any upgrade. Yes, the highlight is not present in both, Discover and Lense created from discover. I tried again from scratch, creating the Data View like I did it the first time.

Note, that I have not never seen the "pop up" you mentioned about the Observability.
The Observability is selected for all the sub options.

Here is my log index mapping:

{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "app_campus": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "app_env": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "app_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "app_version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "elapsed_time": {
        "type": "float"
      },
      "event_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "extended_properties": {
        "properties": {
          "Aurora": {
            "properties": {
              "Eca": {
                "properties": {
                  "ContentRoot": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "EnvName": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "Index": {
                    "properties": {
                      "ContentRoot": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "EnvName": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "Topic": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "address": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "eventName": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "topic": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  },
                  "KeyId": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "address": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "path": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "Eci": {
                "properties": {
                  "Search": {
                    "properties": {
                      "ClientName": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "ContentRoot": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "DisposedCount": {
                        "type": "long"
                      },
                      "ElapsedMilliseconds": {
                        "type": "float"
                      },
                      "EnvName": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "HandlerLifetime": {
                        "type": "float"
                      },
                      "InitialCount": {
                        "type": "long"
                      },
                      "RemainingItems": {
                        "type": "long"
                      },
                      "address": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "eventName": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "topic": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "instance_id": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "log_level": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "message": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

Hi @Matt_Janda,

Can you please try adding xpack.spaces.experimental.forceSolutionVisibility: true to your kibana.yml and then create a new Space with Solution View "Observability"?

You can share your existing data views with this new space on Saved Objects page.

Will do it shortly. Quick question, I am passing all the parameters as environment variables in my compose file. How do I do this one ?

My kibana.yml inside the container is generated automatically:

#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

Hi Julia,

I've followed your hints, created new space and selected "Observability" as the solution view.

I create a view and went to discover. No luck, am afraid. I like the new solution view, yet it does not highlight the log levels.

What index pattern is configured for central-logging data view?

I will check but that might be the answer :slight_smile:

I'm curious on where we can provide this kind of feedback, we recently moved to Elastic Cloud and per default it uses Solution View, we tried to use it for a couple of weeks but the experience was making simple actions take a lot more time and clicks to be done so we reverted back to classic view.

Biggest issues were related to the management side of things, like Index Management, Data View, Dashboards, Visualizations etc.

Yes, it is called central-logging now. I was hoping that a "keyword" logs would have anything to do with the highlighting as when I started building the dashboard my index and data view was called eca-logs. I tried to revert it back, but no pretty highlighting . :frowning:

I know about the format setting where you can configure the colors, but it is not as pretty as the one I saw.

Hi @leandrojmp,

Thanks for your feedback! I passed it over to the team.

In general feedback regarding the navigation can be provided here per solution type:

The data view name should not make a difference but the index pattern might. Can you please open your data view in edit mode (on Stack Management > Data View page for example) and check what index pattern is entered there?

I think I was a bit lucky. :four_leaf_clover: When I was demoing first version of the services, I left it running on original release. I will send you all the details. Here is what I learned so far:

  1. I was running on Kibana 8.16.0 and the highlight works like a charm.
  2. I only changed Kibana to 8.17.4 and the highlights are gone. I probably used NDJSON generated by 8.16.0
  3. Going back to 8.16.0 and recreating containers and volumes, and the log level highlighting is back.

I will try to investigate further, see if running my latest version on 8.16.0 will work.

Thank you for your help.

Hi Julia,

I have finally found this "easter egg" feature. I am calling it since it feels like one. Here are the conditions:

  1. Works only in Kibana 8.16.x. I have tried 8.17.x with no luck. I did not try earlier versions.

  2. The Data View pattern MUST contain the "log" keyword. In my case it is central_log*. If I use central_logging the log level does not get highlighted.

The only wish would be to continue this feature. Right now for presentation value I'll stick to Kibana 8.16.6.

Same here, were on 8.16. 0 and users loved the new colouring of log.level, Just upgraded to 8.18.0 and highlighting of our log.level fields seems gone even though my data views as well as the index pattern names are ended on the word: logs like these:

Any way to bring them back without changing to Observability Solution View mode?