How do I test my condition statement that is not working?

Elastic Stack Logstash
1

This is my preferred filter to capture "error" from the tag that is added in the 'input' as part of filebeat.
This filter fails.

filter {
  if "error" in [tags] {
      grok {
          match => [ 'message', '^.*=%{INT:_nrings}^.*=%{INT:_phone}^.*=%{WORD:_informat}^.*=%{INT:_tries}^.*=%{INT:_callTime}^.*=%{WORD:_newApp}^.*=%{INT:_retryInterval}^.*=%{GREEDYDATA:_initialScript}^.*=%{INT:_applicationData}^# %{GREEDYDATA:DateTime}^#%{WORD:Result}']
      }
  }
}

This was a test to try match something else in the 'if' statement.
This filter fails as well.

filter {
    if [env] == "production" {
        grok {
            match => [ 'message', '^.*=%{INT:_nrings}\n^.*=%{INT:_phone}\n^.*=%{WORD:_informat}\n^.*=%{INT:_tries}\n^.*=%{INT:_callTime}\n^.*=%{WORD:_newApp}\n^.*=%{INT:_retryInterval}\n^.*=%{GREEDYDATA:_initialScript}\n^.*=%{INT:_applicationData}\n^# %{GREEDYDATA:DateTime}\n^#%{WORD:Result}']
        }
    }
}

Thoughts and or ideas please.
Thank you

This is the message from the filebeat logs:

{
       "message" => "_nrings=9\n_phone=15556686920\n_informat=NONE\n_tries=1\n_callTime=0\n_newApp=arcVXML2\n_retryInterval=0\n_initialScript=http://10.3.3.18:8080/scr/vui/techOutbound/17033746\n_applicationData=15740\n# 2020/03/31 11:01:22 \n##Tue Mar 31 11:01:24 2020\n#OutboundRetCode:503 VXML Event: error.com.arc.tel_initiatecall.tel_failure.no_ring_back",
      "@version" => "1",
    "@timestamp" => "2020-04-04T09:35:27.129Z",
          "tags" => [
        [0] "-sip-logs",
        [1] "system",
        [2] "tech",
        [3] "eWel",
        [4] "error",
        [5] "OCS",
        [6] "tech",
        [7] "beats_input_codec_plain_applied"
    ],
         "input" => {
        "type" => "log"
    },
        "fields" => {
        "env" => "production"
    },
           "ecs" => {
        "version" => "1.4.0"
    },
          "host" => {
         "architecture" => "i686",
                   "os" => {
            "platform" => "centos",
             "version" => "6.10 (Final)",
              "family" => "redhat",
                "name" => "CentOS",
              "kernel" => "2.6.32-754.28.1.el6.i686",
            "codename" => "Final"
        },
                   "id" => "eb4b9dc9c495ff39c8cd902300000019",
        "containerized" => false,
                 "name" => "tech_sip",
             "hostname" => "tech02"
    },
         "agent" => {
            "hostname" => "tech02",
                  "id" => "255b4164-512d-4417-ad26-ab878a3e44a8",
             "version" => "7.6.1",
                "name" => "tech_sip",
                "type" => "filebeat",
        "ephemeral_id" => "e6b52d18-8678-4ef2-b83a-7354b4227aee"
    },
           "log" => {
         "flags" => [
            [0] "multiline"
        ],
        "offset" => 0,
          "file" => {
            "path" => "/home/arc/.ISP/Telecom/OCS/called/CDF.010e63cf-d45c-433d-bfe5-1cadeAAAAA"
        }
    },
          "type" => "filebeat"
}
2

I suggest you read this for advice on how to create a grok pattern for a complex string.

You have newlines in your message, so the pattern has to match them. I would start with

        match => [ 'message', '^%{WORD}=%{INT:_nrings}
 %{WORD}=%{INT:_phone}
 %{WORD}=%{GREEDYDATA:_informat}
 %{WORD}=%{INT:_tries}
 %{WORD}=%{INT:_callTime}
 %{WORD}=%{GREEDYDATA:_newApp}
 %{WORD}=%{INT:_retryInterval}
 %{WORD}=%{URI:_initialScript}']

Extend that one line at a time. Personally I would replace the patterns like %{GREEDYDATA:_newApp} with (?<_newApp>[^_]+).

3

Good points, interesting use of GROK {WORD} to ingest the start of the new line.
I had looked at a regex format for filtering values. I will get that a try as well.

Your ideas are something to consider for sure.

I am expecting to see "_grokparsefailure" in my logs when there are grok issues, but I am not seeing this failure at the moment. So my assumption is that the filter is not passing the event to the grok statement.

Firstly, the filter is not catching the event to consider the grok pattern.
I am not sure why the filter is not catching the event.

Thoughts?

Cheers
Thanks

4

Your grok filters are wrapped in conditionals. Maybe the conditions are not being met.

5

Thanks
I have updated the question to ask why my condition statement is not working and how to correct it.

6

I would expect the test for the "error" tag to work. For the other test, you have a [fields][env] field, not an [env] field. Set fields_under_root in filebeat if you want an [env] field.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.