Check for existence of a field / sub-field

[sandeepkanabar]

For my use case, I want to check the existence of a specific field translogid in log message and if it is not present, I want to discard it (Say raise _grokparsefailure). I followed the steps at Event Dependent Configuration but somehow am not able to make this work.

In order to test this out, I generated log data such that out of the 2 messages, first one is invalid i.e doesn't have the field translogid but instead has the tanslogid1.

Input Log: (minimal data shown for simplicity)

2017-01-11 03:17:17,738 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2}
2017-01-11 03:17:27,741 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=IN, remip=222.230.127.162, translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure}

My Logstash.conf is:

input {
  kafka {
    bootstrap_servers => "10.82.135.10:80,10.82.135.11:80,10.82.135.12:80"
    topics => ["eastus-raw-sas-transaction-log"]
    #decorate_events => true
    codec => "json"
    #type => "colp_summary"
  }
}

filter {
        grok {
            match => [ "message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} \[%{GREEDYDATA:thread}\] %{JAVACLASS:className} - \{%{GREEDYDATA:logmsg}\}" ]
        }

        if "_grokparsefailure" not in [tags] {
          if ![logmsg][translogid] {
                mutate { add_tag => "_grokparsefailure" }
          }
        }
}

output {

    stdout {
            codec => "rubydebug"
    }
}

I tried a lot of different combinations like if ![logmsg][translogid], if ![translogid] including all mentioned in the Event Dependent Config doc, but none seem to work. if ~[translogid] throws can't convert nil into String. The Ruby Output shows the grokparsefailure tag present in both the cases while it should be present only for the 1st message in log.

Ruby Output shows:

{
       "severity" => "INFO",
     "@timestamp" => 2017-01-10T21:49:36.615Z,
    "srcHostname" => "abc",
         "logmsg" => "country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2",
       "@version" => "1",
      "className" => "com.foo.SummaryLogAspect",
         "thread" => "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'",
        "message" => "2017-01-11 03:17:17,738 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2}",
       "serverId" => "201",
           "tags" => [
        [0] "_grokparsefailure"
    ],
      "timestamp" => "2017-01-11 03:17:17,738"
}
{
       "severity" => "INFO",
     "@timestamp" => 2017-01-10T21:49:36.615Z,
    "srcHostname" => "abc",
         "logmsg" => "country=IN, remip=222.230.127.162, translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure",
       "@version" => "1",
      "className" => "com.foo.SummaryLogAspect",
         "thread" => "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'",
        "message" => "2017-01-11 03:17:27,741 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=IN, remip=222.230.127.162,  translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure}",
       "serverId" => "201",
           "tags" => [
        [0] "_grokparsefailure"
    ],
      "timestamp" => "2017-01-11 03:17:27,741"
}

I guess I'm missing something quite simple. Can someone point out how can I check for the existence of translogid field?

Thanks.

[sandeepkanabar]

I eventually figured out the solution. Since I had dynamic number of KV pairs, after grok, I was also using KV filter to split the KV pairs in logmsg field.

After the split, I could easily do:

    if "_grokparsefailure" not in [tags] {
      if ![translogid] {
        mutate { add_tag => "_grokparsefailure" }
      }
    }

However the same didn't work before splitting with KV filter. My guess is that in the logmsg or message field, the KV pairs are still strings and not actually KV pairs. Thus, referring to translogid as field inside logmsg or message field was wrong.