I added a new condition in the logstash output. In this new condition I am checking for a _grokparsefailure tag. If true, Logstash sends this record with another index (unmateched-logs- *). In this way, it let me know all the records that grok pattern are rejecting.
output {
if ([Common]) {
elasticsearch {
hosts => ["xxxxx:9200"]
manage_template => false
index => "global-app1-%{+YYYY.MM.dd}"
user => "xxxxx"
password => "xxxxx"
}
}
if ([loginfo]) {
elasticsearch {
hosts => ["xxxxx:9200"]
index => "global-app2-%{+YYYY.MM.dd}"
manage_template => false
user => "xxxxx"
password => "xxxxx"
}
}
if "_grokparsefailure" in [tags] {
elasticsearch {
hosts => ["xxxxx:9200"]
index => "unmatched-logs-%{+YYYY.MM.dd}"
manage_template => false
user => "xxxxx"
password => "xxxxx"
}
}
}
If you know other way to solve this problem, I invite you to share your idea/solution with the community.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.