Extract value from Nested message field - LogStash Kibana Grok Mutate

Elastic Stack Logstash
1

I've configured a FileBeat service to send logs to Kibana via LogStash. FileBeat and LogStash are in different servers while Kibana is inside an AWS ElasticSearch domain. The logs are visible in the Kibana dashboard. In each record, there's a field called message which consists of 10 sub-fields (nested field). What I need to do is, extract values from those sub-fields and present them as separate fields.

I have tried both GROK and MUTATE to achieve this but no progress.

Let's consider one of the sub-field, named conName.

Using MUTATE : Tried putting field name and sub-field name in square brackets, like explained in one of previous answers. I got %{[message][conName]} as the output.

mutate {
 add_field => {
  "custom_field1" => "%{[message][conName]}"
 }
}

Using MUTATE with split: Following documentation, I tried using split to first split the message field into independent 10 fields, did not work either.

mutate {
 split => { "message" => "," }
 add_field =>  { "message1" => "%{message[0]}" }
}

Using GROK

grok {
   match => { "message" => "%{STRING:conName}" }
}

At least tell me which one should I use: MUTATE or GROK?

2

That would have to be "%{[message][0]}" in any version of logstash from the last couple of years.

What does the original [message] field look like?

3

I changed the syntax like you said, now it's showing the whole message field.
Example record of the message field :

2021-04-05 13:32:03.746+0000 | INFO | BRAINWAVE_API | DEV |  |  | 22460 | [scheduling-1] | c.h.w.a.bwService | Fetching results for orgI : 81ge0-de12ff-59jla0 orgN : Cactus Corp
4

You are splitting the message field using ",", but it is separated with "|". Try

mutate { split => { "message" => "|" } }
1 Like
5

Thank you very much. Replacing the "," with "|" solved the issue.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.