How to parse fields from multiline text log file?

Elastic Stack Logstash
1

Dears,

I need your advise in case of parsing fields from text log file. This is sample of text file:

---- got a mess at: 11:21:51
ctrl_mess_handler::control_message_handler @333.333
got new timer id: 11111, to be fired off in: 2 seconds
ip_timer::cc_timer_expd @333.333
:profiler: 0
setting next timer event in 2 seconds
wa at 11:21:51, q id: 44444
ip_timer::cc_timer_expd @112156.1300527189
sending timer expired msg to q: 98323 for cc_con
SENT OK: timer expired msg to q: 98323, type=0, type2=0
:profiler - wa end: 111
ccad_timer mess receive interrupted, errno = EINTR (sleep on a full mess q condition, the process caught a signal)
:profiler: 111
setting next timer event in 7 seconds
wa at 11:21:56, q id: 44444
:profiler - wa end: 1

What is the best way to extract some fields to separate columns? How to do it?
For example I need these below fields in separate columns in ELK:
"got new timer id: 11111"
"q id: 44444"
"type=0"
"wa end: 1"
Best Regrads,
Dan

2

You could use grok

grok {
    break_on_match => false
    match => {
        "message" => [
            "got new timer id: %{INT:timerId}",
            "q id: %{INT:qId}",
            "type=%{INT:type}",
            "wa end: %{INT:waEnd}"
        ]
    }
}
1 Like
3

Hello @Badger ,

On the base of your answer I prepared some grok patter:

(?m)%{TIME}%{GREEDYDATA}got new timer id: %{INT:timerId}%{GREEDYDATA}q id: %{INT:qId}%{GREEDYDATA}type=%{INT:type}%{GREEDYDATA}wa end: %{INT:waEnd}

and it looks good. I'll test your proposal too. Thanks a lot.

Best Regards,
Dan

4

Having a pattern with several GREEDYDATA embedded is going to be more expensive than using multiple patterns. Maybe not expensive enough to matter, but at least a little.

5

Hello @Badger,

Thanks a lot for your explanation. I'll test your proposed sollution.

Best Regards,
Dan

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.