Using Logstash, not able to filter Error Level while indexing to elastic search

[Rakesh_Katakam]

Hi,

I recently started using ELK stack and we came across a scenario where to filter Error level logs in application log files and want to push Elastic search to do more visualization on logs using Kibana.

Here below is the Event pipeline which we have written to extract logs and push to elastic search.

input {
   file {
      path => "D:/obe.log"
	  codec => multiline {
      pattern => "<%{LOGLEVEL:log-level}>%{SPACE}<%{TIMESTAMP_ISO8601:timestamp}>%{SPACE}<%{NUMBER:PID}>%{SPACE}<%{GREEDYDATA:processName}>%{SPACE}<%{JAVACLASS:class}>%{SPACE}<%{JAVALOGMESSAGE:logmessage}>"
      negate => true
      what => "previous"
    }
   }
}
output {
	if [log-level] == "ERROR" {
		elasticsearch {
			hosts => ["127.0.0.1:9200"]
			index => "errors"
		}
		stdout {}	
	}	
}

In output plugin, with out if condition, I observed all logs got indexed to elastic search which is working as expected, but by adding if [log-level] == "ERROR", I don't see any events getting indexed.

Could you please suggest if there is any mistake in the configuration which I have shown above (or) if there is any other way..

Thanks,
Rakesh.

[Rakesh_Katakam]

It got resolved by doing configuration in below way..

input {
   file {
      path => "D:/obe.log"
	  codec => multiline {
      pattern => "^<%{LOGLEVEL:log-level}>"
      negate => true
      what => "previous"
    }
   }
}
filter{
    grok{
        match => {"message" => "<%{LOGLEVEL:log-level}>%{SPACE}<%{TIMESTAMP_ISO8601:timestamp}>%{SPACE}<%{NUMBER:PID}>%{SPACE}<%{GREEDYDATA:processName}>%{SPACE}<%{JAVACLASS:class}>%{SPACE}<%{JAVALOGMESSAGE:logmessage}>"}
    }
}	
output {
	if [log-level] == "ERROR" {
		elasticsearch {
			hosts => ["127.0.0.1:9200"]
			index => "errors"
		}
		stdout {}	
	}	
}