# Check for existence of a field / sub-field

**URL:** <https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160>\
**Category:** Logstash\
**Created:** [January 10, 2017, 10:12pm UTC](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160 "2017-01-10T22:12:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [January 10, 2017, 10:12pm UTC](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160/1 "2017-01-10T22:12:59Z")

</div>

For my use case, I want to check the existence of a specific field `translogid` in log message and if it is not present, I want to discard it (Say raise `_grokparsefailure`). I followed the steps at [Event Dependent Configuration](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) but somehow am not able to make this work.

In order to test this out, I generated log data such that out of the 2 messages, first one is invalid i.e doesn't have the field `translogid` but instead has the `tanslogid1`.

Input Log: (minimal data shown for simplicity)

```
2017-01-11 03:17:17,738 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2}
2017-01-11 03:17:27,741 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=IN, remip=222.230.127.162, translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure}

```

My Logstash.conf is:

```
input {
  kafka {
    bootstrap_servers => "10.82.135.10:80,10.82.135.11:80,10.82.135.12:80"
    topics => ["eastus-raw-sas-transaction-log"]
    #decorate_events => true
    codec => "json"
    #type => "colp_summary"
  }
}

filter {
        grok {
            match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} \[%{GREEDYDATA:thread}\] %{JAVACLASS:className} - \{%{GREEDYDATA:logmsg}\}" ]
        }

        if "_grokparsefailure" not in [tags] {
          if ![logmsg][translogid] {
                mutate { add_tag => "_grokparsefailure" }
          }
        }
}

output {

    stdout {
            codec => "rubydebug"
    }
}

```

I tried a lot of different combinations like `if ![logmsg][translogid]`, `if ![translogid]` including all mentioned in the Event Dependent Config doc, but none seem to work. `if ~[translogid]` throws `can't convert nil into String`. The Ruby Output shows the `grokparsefailure` tag present in both the cases while it should be present only for the 1st message in log.

Ruby Output shows:

```
{
       "severity" => "INFO",
     "@timestamp" => 2017-01-10T21:49:36.615Z,
    "srcHostname" => "abc",
         "logmsg" => "country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2",
       "@version" => "1",
      "className" => "com.foo.SummaryLogAspect",
         "thread" => "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'",
        "message" => "2017-01-11 03:17:17,738 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=JP, remip=222.230.107.165, tanslogid1=aca89691-60f6-4dc8-b994-9808187798fb, srcType=INT2}",
       "serverId" => "201",
           "tags" => [
        [0] "_grokparsefailure"
    ],
      "timestamp" => "2017-01-11 03:17:17,738"
}
{
       "severity" => "INFO",
     "@timestamp" => 2017-01-10T21:49:36.615Z,
    "srcHostname" => "abc",
         "logmsg" => "country=IN, remip=222.230.127.162, translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure",
       "@version" => "1",
      "className" => "com.foo.SummaryLogAspect",
         "thread" => "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'",
        "message" => "2017-01-11 03:17:27,741 INFO [[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'] com.foo.SummaryLogAspect - {country=IN, remip=222.230.127.162, translogid=986813f6-c732-48b7-b6d1-31c420e8cb30, srcType=Azure}",
       "serverId" => "201",
           "tags" => [
        [0] "_grokparsefailure"
    ],
      "timestamp" => "2017-01-11 03:17:27,741"
}

```

I guess I'm missing something quite simple. Can someone point out how can I check for the existence of `translogid` field?

Thanks.

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [January 11, 2017, 12:15am UTC](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160/2 "2017-01-11T00:15:52Z")

</div>

I eventually figured out the solution. Since I had dynamic number of KV pairs, after grok, I was also using KV filter to split the KV pairs in `logmsg` field.

After the split, I could easily do:

```
    if "_grokparsefailure" not in [tags] {
      if ![translogid] {
        mutate { add_tag => "_grokparsefailure" }
      }
    }

```

However the same didn't work before splitting with KV filter. My guess is that in the `logmsg` or `message` field, the KV pairs are still `strings` and not _actually_ `KV pairs`. Thus, referring to `translogid` as field inside `logmsg` or `message` field was wrong.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2017, 12:16am UTC](https://discuss.elastic.co/t/check-for-existence-of-a-field-sub-field/71160/3 "2017-02-08T00:16:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.