Display raw JSON value from log entry in Discover table

[Marcin_Kawa]

I'm looking for a way to combine few fields from log entry into one string representation to display it on the Discover view. Problem is the only common element for the fields is that those values are enclosed inside JSON object extra.

Example log entry JSON:

{
   "message": "MainProcess",
   "extra": {
      "task_name": "Indexing Sub Plan Bundles          ",
      "function": "<function load_bundles>",
      "memory_used": 519480,
      "time_sec": 27.590729
    },
}

So in Kibana table view it's:

message: "MainProcess",
extra.task_name: "Indexing Sub Plan Bundles ",
extra.function: "<function load_bundles>",
extra.memory_used: 519480,
extra.time_sec: 27.590729

Is there a way to display the whole extra key as string on the list in the Discover view?

Thanks. Any help would be appreciated!

[weltenwort (Felix Stürmer)]

Hi @Marcin_Kawa,

one option would be to add the string representation in a new field during ingest. If that is not an option, you could create a scripted field in the index pattern settings that concatenates the fields you want dynamically. But please be aware of the performance penalty associated with that.

I would like to understand your use case better, if you don't mind. What prohibits you from just adding those columns to the table individually in Discover?

[Marcin_Kawa]

Hi @weltenwort,
Many thanks for the reply. During ingest , you mean when Logstash processes the entry? It would duplicate the data though, right?

For scripting fields the problem is that the extra object keys can differ a lot between log entries. So for example in one entry log there will be 5 different keys in extra object when the next log entry can have 10 totally different keys inside the extra object.

# log entry 1
{"message": "foo", "extra": {"bar": 'one', "foobar": "two"}}
# log entry 2
{"message": "foo", "extra": {"barbar": 'bar', "foofoo": "foo"}}
# log entry 3
{"message": "foobar"}

The reason I want to display content from extra is to see what's inside (keys and values) without expanding each entry. Sometimes the extra object contains a lot of data so it would be better not o replicate it in the log entry as it would be used only for display purposes in the Discover list view.

I believe the scripting field would not work here as I want to display all keys/values from the extra object and the keys inside are inconsistent between different log entries.

[weltenwort (Felix Stürmer)]

There indeed is no way to iterate over all fields of a document in a scripted field. Aside from duplicating the data in a string field on ingest time, I can not see another way of achieving this right now, sorry.

But I would encourage you to file an enhancement request on GitHub any explain your use case. This way we can track the request and possibly implement a solution down the line.

[Marcin_Kawa]

Many thanks for help. I will do create enhancement request as I believe that display only fields might be useful. For now I'm duplicating the extra argument and parsing it to string with mutate filter in Logstash.