I am using filebeat to transfer files to logstash. In logstash I am using the following configuration:
filter {
grok {
match => { "message" => "%{DATESTAMP:date1}%{SPACE}%{LOGLEVEL:severity}%{SPACE}[(?[a-zA-Z0-9-/:#\s[]]*)]" }
tag_on_failure => [ "grok1" ]
}
}
The file transmitted contains over 1000 lines, but in Kibana I can only see 150 hits in total. If I reduce the match filter to:
match => { "message" => "%{DATESTAMP:date1}%{SPACE}%{LOGLEVEL:severity}%{SPACE}"}
then I all "lines" are displayed in kibana.
In Kibana I'd like to see the messages for which the grok filter failed with a corresponding tag "grok1" but this is not the case. That would help to debug the filter itself ...