Best way to parse multiple message patterns

[Thuunder7]

Hello,

I want to ask if it is possible to have multiple dissect patterns?
I know i can create conditionals based on the "_dissectfailure" and create another dissect to parse other patterns, but this doesn't prevent the previous dissect to print a warning message.
I am currently facing an issue while using dissect { mapping => ... }, when the field message contains a pattern different than the one i defined it always prints a warning message on the logstash logs. Is there a way to silence this logs?

Regards,

[Rios]

I think is it's not possible because there is no match plugin as in grok. You can:
a) Use IF

add if fielda=="value1" {
 dissect {
    mapping => {"field1 filed 2...."}
}
else if fieldb=="value1" {
 dissect {
    mapping => {"field1 filed 2...."}
}

Field A anf B can get by grok or similar .
b) use field count, then if... else base on field numbers.

[Thuunder7]

Yes but sadly i don't have a field to make a flow like you did. That would be the best approach.
Do you know if GROK is more silent than Dissect? Since i don't have a field to make IFs, i would like to at least silence these logs.
I have heard that dissect is more agressive and it will always print a log on the console.

[Rios]

Can you provide few lines as sample? If something is classified just replace with dummy data. Need to see fields structure.

[Thuunder7]

The pattern that i am using is the following:

if [metadata][kafka][topic] == "my-log-example"
  dissect { mapping => {
    "message" => "[%{}][%{}][%{[log][type]}]%{}: %{}: %{[log][message]}"
  }

I have logs which doesn't have the above pattern , so every time they reach this dissect logstash will print warning logs... I treat these logs later but i would like to silence this warning logs from logstash.

Example of a log without the correct pattern:

[][evtlog][warn] Queue overflow: 310 events lost

Example of a log with correct pattern:

[xxxxxx][xxxxxx][log][notice] mpgw(xxxxxxxxxxxxx): trans(11111111)[response][x.x.x.x] gtid(1111111111111111): more_info_here

[Rios]

Yes, shouldn't be problem to implement with grok, multi match.

      grok {
        patterns_dir => "./patterns"
        match => {"message" => [ "%{PATTERN1}",  "%{PATTERN2}" ] }
      } 

[leandrojmp (Leandro Pereira)]

With grok you can have multiple patterns to match in the same filter.

With dissect you can only have one pattern per filter, so the best approach is to filter the type of message and direct it to the correct dissect, you don't need a specific field for this, but you need some pattern in a field.

For example, you shared two types of message, if the one that does not match your pattern always have this string Queue overflow, you would be able to filter it out with the following:

if "Queue overflow" in [message] { do something }

But if your only issue is silencing the Dissect filter, you can change the log level.

Run the following in your logstash server and it will stop logging the dissect failure warnings, will log only if the filter has an error.

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'{ "logger.org.logstash.dissect.Dissector" : "ERROR" }'

[Thuunder7]

I have tried to use the first approach:

if "Queue overflow" in [message] { do something }

And it works perfectly.

Thank you Leandro and Rios