Extracting a new field from the message field

Elastic Stack Logstash
1

2020-03-06 09:36:52.801907464 re0:ndp:25786 lltp_debug message = "NDP-DBG:NC_FSM_HANDLER:1634:: state 7, event 4, NexthopId 44001"

This is my log. I need to extract state and NexthopId from the message field

match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype}.%{mssg:Message}."}

custom pattern
mssg ((Msg|message|Message|message1|message2) [=] ["]%{DATA}+["])

2

Hi there,

do you need to extract only the NexthopId or all the other fields you put in your grok, too?

3

Anyway, what about this:

filter {
  grok {
    break_on_match => false
    pattern_definitions => { "mssg" => "(Msg|message|Message|message1|message2) [=]" }
    match => {
      "message" => ["%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} \"%{GREEDYDATA:Message}\""]
      "Message" => ["NexthopId %{WORD:next_hop_id}"]
    } 
  }
}

Obviously, if you're not sure you can to extract the Message field from every event, you can make two grok filters and put the second one (the one on Message) in a condition like if [Message] { ...grok filter...}.

Also, just check if there's any event with a different pattern which breaks the first grok. That is up to you.

4

Hi Fabio,

I tried your filter. But when i run logstash, it uses the default mapping template. There is no error in the conf file. Can you help me out

And I am new to logstash :slight_smile:

5

This is my config file

input {

file {
path => ["/home/hari/ndppro/ndp2a.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
}

}
filter {

grok
{
patterns_dir => ["/etc/logstash/pattern"]
match => {
"message" => ["%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} "%{GREEDYDATA:Message}""]
"Message" => ["NexthopId %{WORD:next_hop_id}"]
}
}
mutate
{
remove_field => [ "message" ]
}

}
output {
elasticsearch{
hosts => ["localhost:9200"]
}
}
~

6

So, first of all when posting some code (or anything which is not plain writing) please format it, or it'll be impossible to read for us.
So use any editor (VSCode, Atom, Sublime or whatever) to properly indent your code, paste it here properly indented, highlight it and click on the Preformatted tool (image ).

Speaking of your question, I don't really get what you mean by

it uses the default mapping template

Plus, if I insert something in my pipeline (like the break_on_match => false in the grok, why did you remove it?

Finally, can you post here some outputs of the following pipeline (output will be in your standard output, so your terminal):

input {
  file {
    path => ["/home/hari/ndppro/ndp2a.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    break_on_match => false
    pattern_definitions => { "mssg" => "(Msg|message|Message|message1|message2) [=]" }
    match => {
      "message" => ["%{TIMESTAMP_ISO8601:timestamp} %{WORD:node}:%{WORD:program}:%{INT:pid} %{WORD:tracetype} %{mssg} "%{GREEDYDATA:Message}""]
      "Message" => ["NexthopId %{WORD:next_hop_id}"]
    }
  }

  mutate {
    remove_field => [ "message" ]
  }
}

output {
  stdout{}
}
7

Hi Fabio,

Thanks. I made some mistake. It works fine now. :smiley:

Now i have a scenario where i must add a field to display the state name for the corresponding state number.

Please find my log below
2020-03-06 09:36:50.775744749 re0:ndp:25786 lltp_debug message = "NDP-DBG:NC_FSM_HANDLER:1634:: state 4, event 0, NexthopId 44000"
2020-03-06 09:36:51.548239404 re0:ndp:25786 lltp_debug message = "NDP-DBG:NC_FSM_HANDLER:1634:: state 5, event 3, NexthopId 44001"
2020-03-06 09:36:52.778379389 re0:ndp:25786 lltp_debug message = "NDP-DBG:NC_FSM_HANDLER:1634:: state 5, event 0, NexthopId 44001"

States
0 - No state
1 - Unreachable
2 - Incomplete
3 - Reachable
4 - Stale
5 - Delay
6 - Probe

Can you help me on how to write the conditions?

8

Glad it works properly,

I do not have access to my laptop right now but I can tell you the steps to follow:

  • extract the state from the Message field using a grok again with a break_on_match: false

  • use a translate filter or a ruby filter. The translate one is kinda trivial to use (it's basically a dictionary) and you can see an example on the logstash documentation.

9

Thanks Fabio.

I'll have a look into it.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.