Logstash multiline help

Elastic Stack Logstash
1

I am having an issue cobining this log into a multiline. Any suggestions on how I can use a specific timestamp is the identified to combine the line?

Log Example (notice the timestamp is the same):
[Thr 47613344799040] Trace dispatcher thread is successfully triggered [Thr 47613344799040] NCS data timer thread is successfully triggered [Thr 47613344799040] Data dispatcher thread is successfully triggered [Thr 47613344799040] Current NCS Configuration: [Thr 47613344799040] trace_enabled_via_e2e=1; [Thr 47613344799040] trace_enabled_via_em=0; [Thr 47613344799040] trace_preview_timeout=120; [Thr 47613344799040] tracecheck_interval=5; [Thr 47613344799040] tracesize_threshold=1000000; [Thr 47613344799040] zombie_check_interval=300; [Thr 47613344799040] zombie_timeout=86400; [Thr 47613344799040] NCS configuration info end----------------

Is there any way to use match the timestamp to combine the line?

2

You might have the same problem that I have : Combining the lines will not produce what you expect.
My post.
I posted the question today, but I'm still trying to figure it by myself or waiting for a miracle :slightly_smiling:
By the way, you will easily find what you need to determine the pattern for the multiline codec on that page :
Github-grok patterns

3

Thanks ! Your issue is a bit different. I posted my solution to what I think you trouble is. For my issue, I want to combine lines based on timestamp x, rather than just matching the pattern. I guess it would be like matching anything in a log file with process id x, and make it multiline.

This may not be possible, but I wanted to ask the experts out there.

4

If it's possible, it may help me too for my issue (the solution you proposed me just made the reverse effect : everything is linked to these particular logs => Because they are those who match the pattern. But thanks for the try ! )
I keep an eye on your post, just in case :wink:

5

I figured it out. I guess my original google search was too specific.

grok { match => [ "message", "\[%{WORD} %{NUMBER:TSID}\]" ] } multiline { stream_identity => "%{TSID}" pattern => "." # match anything because we're gathering by id field what => "previous" periodic_flush => true max_age => 10 # however many seconds it takes to get all of your lines together }

This results in the log above being combined into a multiline.

Found here:
http://stackoverflow.com/questions/32304023/how-to-make-logstash-multiline-filter-merge-lines-based-on-some-dynamic-field-va

1 Like
6

That's nice : but multiline filter is deprecated... And "stream_identity" doesn't exist in multiline codec :confused:
I search for the equivalent, and I warn you if I find anything