New fields are not added while grok seems fine


(Yonatan Brand) #1

Hi,

I am not able to see new fields I created with grok. I have tried to refresh the indexes, however, I am still not able to see them.

Below is my logstash configuration:

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}

filter {
    if [log_type] == "artifactory" {
            grok {
                    match => ["message", ["^%{TIMESTAMP_ISO8601:artifactory_timestamp} \[%{DATA:artifactory_thread_name}] \[%{LOGLEVEL:loglevel}%{SPACE}] \(%{DATA:artifactory_event_type}\) - %{GREEDYDATA:artifactory_message}"]]
            }
    }

    else if [log_type] == "request" {
            grok {
                    match => ["message", ["%{DATA:artifactory_timestamp}\|%{NUMBER:artifactory_request_time}\|%{DATA:artifactory_request_type}\|%{IP:artifactory_client_ip}\|%{DATA:artifactory_username}\|%{DATA:artifactory_request_method}\|%{DATA:artifactory_resource_path}\|%{DATA:artifactory_protocol_version}\|%{DATA:artifactory_response_code}\|%{GREEDYDATA:artifactory_request_content_length}"]]
            }
    }

mutate {
    copy => { "[fields][log_type]" => "log_type" }
    }
}

output {

    if [log_type] == "request"{
            elasticsearch {
                    hosts => "10.128.0.243:9200"
                    index => "request-%{+YYYY.MM.dd}"
            }
    }

    else if [log_type] == "artifactory"{
            elasticsearch {
                    hosts => "10.128.0.243:9200"
                    index => "artifactory-%{+YYYY.MM.dd}"
            }
    }
}

Below are two log examples to verify grok is ok:

  1. Request
    20181106155947|17|REQUEST|10.128.0.207|admin|GET|/api/system/licenses|HTTP/1.1|200|0

  2. Artifactory
    2018-11-06 15:40:41,091 [http-nio-8081-exec-9] [INFO ] (o.a.s.ArtifactoryApplicationContext:495) - Artifactory application context set to READY by reload

Any assistance is much appriciated.

Thanks!


(Christian Dahlqvist) #2

Filters are executed in order, and it looks to me like you are only creating the log_type field after you have tried to use it in conditionals. This means that your grok filters will never be run as the field does not exist at that point.


(Yonatan Brand) #3

Hi Christian,

Thank you for your reply.

So foolish of me to miss it. Probably I was to tired :tired_face:.

My issue is resolved.

Thank you.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.