How Can I find unmatched logs with pattern grok?

[JaviiPG (Javi)]

Hi everybody,

I has been configured a ELK environment. It has working properly .

I have a grok filter that is valid for all APP logs. However i want be sure that all logs are adding to Elasticsearch.

Are there any way I can identify records that do not pass the grok pattern? I would like identify these logs to be sure all logs are being handled.

Could I insert these logs in other index? How Can do it?

Thanks a lot and best regards!!

[JaviiPG (Javi)]

I just applied a possible solution to my problem.

I added a new condition in the logstash output. In this new condition I am checking for a _grokparsefailure tag. If true, Logstash sends this record with another index (unmateched-logs- *). In this way, it let me know all the records that grok pattern are rejecting.

output {
    if ([Common]) {
            elasticsearch {
                    hosts => ["xxxxx:9200"]
                    manage_template => false
                    index => "global-app1-%{+YYYY.MM.dd}"
                    user => "xxxxx"
                    password => "xxxxx"
            }
    }
    if ([loginfo]) {
            elasticsearch {
                    hosts => ["xxxxx:9200"]
                    index => "global-app2-%{+YYYY.MM.dd}"
                    manage_template => false
                    user => "xxxxx"
                    password => "xxxxx"
            }
    }
    if "_grokparsefailure" in [tags] {
            elasticsearch {
                    hosts => ["xxxxx:9200"]
                    index => "unmatched-logs-%{+YYYY.MM.dd}"
                    manage_template => false
                    user => "xxxxx"
                    password => "xxxxx"
            }

    }

}

If you know other way to solve this problem, I invite you to share your idea/solution with the community.

Best regards!!!
Javi