Custom match pattern based on log message type

[sergle]

Hey there,

Would very much appreciate some input on something I am trying to achieve.

The setup:

NLog -> (UDP) -> Logstash (ELK)

The NLog is configured to send different message structures to Logstash, all in the form of:

<date><msg_result><msg_checksum><msg_payload>. The one thing that is constant on all messages is the message header structure which is <date><msg_result><msg_checksum>. The <msg_payload> varies what it holds.

The grok is configured as follows:

grok {
      break_on_match => true
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}>"
      }
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}>"
      }
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}><%{DATA:SomeMoreData}>"
      }
    }

Will grok only match the first part of the message (message header) and ignore the other since they all contain the same header <date><msg_result><msg_checksum>? The second question, how does one apply filter or match pattern based on <msg_result> before trying to match the payload?

I guess what I would like to know is how does grok digests and matches patters.

I have found somewhat similar situation to what I am trying to achieve here: Use field value to match custom Grok pattern

Thanks a lot

[Badger]

If you supply an option multiple times then logstash will combine them. It usually does that in the way you should expect, but sometimes not. As a result, I never use multiple instances of an option. I suggest you try

  match => {
      "message" => [
          "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}><%{DATA:SomeMoreData}>",
          "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}>",
          "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}>"
      ]
  }

That will try to match the first pattern first, then move on to the second if the first fails to match, then try the third.