Poorly structured log file


Hello All,
I am trying to analyze a log file with the logstash-elasticsearch-kibana setup.
I am still a newby about these nice products and I have a big doubt.

The kind of files I will be working with have very different event structures
(there are dozens, maybe hundreds of kinds of events), some of the events are multiline, some not, some contain JAVA code, some contain lists of addresses, some just lines of asterisks, etc...

As far as I understand the configuration of logstash (with grok for example) has to be very precise and even a space mismatch can stop the pipeline. Is there a (kind of easy) way to handle a situation like mine?

Do you have any suggestion?

thank you,

(Magnus B├Ąck) #2

You can specify multiple grok patterns and have Logstash pick the first one that matches.

Space mismatches won't stop the pipeline. The grok filter will tag the message with _grokparsefailure and pass it on.

(system) #3