Logstash : parse multiple lines, each in one field in a single event

Elastic Stack Logstash
1

Hi everyone,

According to this thread :Multiline conf file to parse log file to elasticsearch - #5 by michaellizhou

The file input isn't meant to read entire files so you're sort of fighting the system here. I suggest you configure multiline to join the current line with the previous unless it matches the finish line. That should get Logstash to emit a logical line containing the whole files once it reaches the "test_web_events.py: FINISH: ..." line. I guess you won't get that line included in the message but that's probably not a problem.

He helped someone who wanted to parse multiple lines in a single event, and at this end he told him :
" I guess you won't get that line included in the message but that's probably not a problem."

But actually, it's my problem, I have all my lines in a single event but after that I want to put each line in a single field in the same event.

OccuredAt: 6/23/2018 01:24:04 AM
Message: Erreur lors du chargement de la DA : 125454
Location: (-1): .
LogLevel: Error
ManagedThreadId: 0
Logname: Ir.Logging
Username: SV_IS-SERV-DV
Machinename: DVMI03
AppDomain:
Identity:

and i give you my conf file :

OccuredAt: %{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{TIME} (?:AM|PM)
Message: %{DATA:message}
Location: %{DATA:location}
LogLevel: %{NOTSPACE:loglevel}
ManagedThreadId: %{DATA:threadid}
Logname: %{NOTSPACE:logname}
Username: %{NOTSPACE:username}
Machinename: %{NOTSPACE:machinename}
AppDomain: %{DATA:appdomain}
Identity: %{DATA:identity}

Even after validate my pattern with Grokdebugger, it doesn't work when i used it in my config file. All the lines of this event are in the same message field but i can't be filtered correctly.

Please, can you explain how to do this, I can't figured it out.

Thanks a lot for your help,

Jérémie

2

There are several things that do not match in that pattern. "Machine Name" vs. "Machinename". "Log Name" vs. "Logname". "SV_IS-SERV-DV" does not match "VINCIC-FR\%{NOTSPACE:username}". And that \ needs to be escaped or you will get a syntax error.

I suggest you start small with something like

    grok {
        match => [ "message", "OccuredAt: %{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{TIME} (?:AM|PM)
Message: %{DATA:message}
" ]

then add one line to the grok pattern at a time and make sure each addition to the pattern works.

1 Like
3

Hi,

Thanks for your answer.

Even with your changes it does not work. I already try all these little possibilities before asking help to someone.

But I found the dissect filter that will help me to parse my data. https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-dissect.html

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.