Custom match pattern based on log message type

Hey there,

Would very much appreciate some input on something I am trying to achieve.

The setup:

NLog -> (UDP) -> Logstash (ELK)

The NLog is configured to send different message structures to Logstash, all in the form of:

<date><msg_result><msg_checksum><msg_payload>. The one thing that is constant on all messages is the message header structure which is <date><msg_result><msg_checksum>. The <msg_payload> varies what it holds.

The grok is configured as follows:

grok {
      break_on_match => true
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}>"
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}>"
      match => {
        "message" => "<%{TIMESTAMP_ISO8601:Date}><%{NUMBER:ProcessId}>%<{NUMBER:Result}><%{NUMBER:Checksum}><%{DATA:SomeData}><%{DATA:SomeMoreData}>"

Will grok only match the first part of the message (message header) and ignore the other since they all contain the same header <date><msg_result><msg_checksum>? The second question, how does one apply filter or match pattern based on <msg_result> before trying to match the payload?

I guess what I would like to know is how does grok digests and matches patters.

I have found somewhat similar situation to what I am trying to achieve here: Use field value to match custom Grok pattern

Thanks a lot

If you supply an option multiple times then logstash will combine them. It usually does that in the way you should expect, but sometimes not. As a result, I never use multiple instances of an option. I suggest you try

  match => {
      "message" => [

That will try to match the first pattern first, then move on to the second if the first fails to match, then try the third.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.