New fields are not added while grok seems fine

[Yonatan_Brand]

Hi,

I am not able to see new fields I created with grok. I have tried to refresh the indexes, however, I am still not able to see them.

Below is my logstash configuration:

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}

filter {
    if [log_type] == "artifactory" {
            grok {
                    match => ["message", ["^%{TIMESTAMP_ISO8601:artifactory_timestamp} \[%{DATA:artifactory_thread_name}] \[%{LOGLEVEL:loglevel}%{SPACE}] \(%{DATA:artifactory_event_type}\) - %{GREEDYDATA:artifactory_message}"]]
            }
    }

    else if [log_type] == "request" {
            grok {
                    match => ["message", ["%{DATA:artifactory_timestamp}\|%{NUMBER:artifactory_request_time}\|%{DATA:artifactory_request_type}\|%{IP:artifactory_client_ip}\|%{DATA:artifactory_username}\|%{DATA:artifactory_request_method}\|%{DATA:artifactory_resource_path}\|%{DATA:artifactory_protocol_version}\|%{DATA:artifactory_response_code}\|%{GREEDYDATA:artifactory_request_content_length}"]]
            }
    }

mutate {
    copy => { "[fields][log_type]" => "log_type" }
    }
}

output {

    if [log_type] == "request"{
            elasticsearch {
                    hosts => "10.128.0.243:9200"
                    index => "request-%{+YYYY.MM.dd}"
            }
    }

    else if [log_type] == "artifactory"{
            elasticsearch {
                    hosts => "10.128.0.243:9200"
                    index => "artifactory-%{+YYYY.MM.dd}"
            }
    }
}

Below are two log examples to verify grok is ok:

1. Request
   20181106155947|17|REQUEST|10.128.0.207|admin|GET|/api/system/licenses|HTTP/1.1|200|0

2. Artifactory
   2018-11-06 15:40:41,091 [http-nio-8081-exec-9] [INFO ] (o.a.s.ArtifactoryApplicationContext:495) - Artifactory application context set to READY by reload

Any assistance is much appriciated.

Thanks!

[Christian_Dahlqvist]

Filters are executed in order, and it looks to me like you are only creating the log_type field after you have tried to use it in conditionals. This means that your grok filters will never be run as the field does not exist at that point.

[Yonatan_Brand]

Hi Christian,

Thank you for your reply.

So foolish of me to miss it. Probably I was to tired .

My issue is resolved.

Thank you.