HI!
I have some huge multiline logs from vendor programm with different info on each line. All lines have several patterns to match. Almost each line starts with INFO\DEBUG word after that it has some datetime numbers pid of process and than finally some important or meaningless info. In one line there is some id's in others there are timestamp of starting or ending event. So I want to parse it that way so i can find in elastic by id of events duration of some processes that are shown in log and so on. Is there a way to I can do that with logstash?
I've tried elapsed plugin already. I can get some timing by using it but can't add id's because it's in different lines.
Now i trying multiline codec but it seems to be not that i want because it just store all lines in one field and if i use something like
"if grokeparserfailure" and than add tag
so I'll have very much to compute.
I have aroung 100G logs\day.
Logs example:
DEBUG 2018/08/14 12:30:03.389 58830 Start event*
...
INFO 2018/08/14 12:30:03.389 58830 Receive new message*
INFO 2018/08/14 12:30:03.389 58830 Message found*
DEBUG 2018/08/14 12:30:03.393 58830 About to Process my message*
INFO 2018/08/14 12:30:03.393 58830 keyword1Start process time that i need to calc*
...
DEBUG 2018/08/14 12:30:03.395 58830 Many meaningless lines
...
INFO 2018/08/14 12:30:03.398 58830 keyword2Exit process*[%Some uniq id here%]
...
DEBUG 2018/08/14 12:30:03.399 58830 400 more lines
...
INFO 2018/08/14 12:30:03.507 58830 Work Commited*
INFO 2018/08/14 12:30:03.507 58830 Stop event*
...
DEBUG 2018/08/14 12:30:04.250 58830 Start event*
... and so on
From this log i want to have one event with tags like
UID => uniq id
EventStart => 2018/08/14 12:30:03.389
ProcTime => 0.005 sec