How to grok complex patterns with logstash 7.1

Elastic Stack Logstash
11

Your dissect pattern does not match the message. You have a very complicated pattern, so I would develop it one field at a time. Start by matching the first field on the line. Do not try to match anything more than that. Start a copy of logstash with

--config.reload.automatic

enabled. That way you only pay the startup cost once, and it will reload the configuration and reinvoke the pipeline each time you modify the configuration. Create a file that contains your log file line then start with

input { 
    file {
        path => "C:/some/path/log.txt"
        start_position => "beginning"
        sincedb_path => "nul"
    }
}
filter { dissect { mapping => { "message" => "####<%{DateTime}>%{}" } } }
output { stdout { codec => rubydebug } }

The trailing %{} in the pattern is needed to consume (and discard) the rest of the message. Once you see a good value for DateTime in the rubydebug output, add the next field to the pattern (or two if you feel lucky). Once you write out the configuration file from your editor logstash will notice a couple of seconds later and restart the pipeline and print out a rubydebug event with (hopefully) some more fields on the event. Repeat this as you perfect the pattern for each field.

It may sound like a lot of effort, but it is actually much quicker than staring at a complicated pattern that doesn't quite match. This way you only ever have to worry about extending the match by one field.

If you end up having to go back to grok then this post describes a similar technique for grok.

1 Like