Logstash does not identify ERROR log level

Elastic Stack Logstash
1

I have a config file like this:

input{
    file{
        path => ["/home/formcept/fcstack/fclogs/relax/logging.log"]
        start_position => "beginning"
        codec => multiline
                    {
                                pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY}"
                                negate => true
                                what => "previous"
                        }
    }
}
filter{
    grok{
        match => {"message" => "%{GREEDYDATA:log_date} %{TIME:log_time} %{WORD:log_level}  %{DATA:log_class} - %{GREEDYDATA:log_message}"}
    }
    mutate{
        add_field => {"log_timestamp" => "%{log_date} %{log_time}"}
                remove_field => ["message","path","log_date","log_time"]
    }
    date{
        match => [ "log_timestamp", "yyyy-mm-dd HH:mm:ss" ]
          target=>"log_timestamp"
    }
}
output {
         stdout {codec => rubydebug}
    }

It works fine but it shows a grokparsefailure error for the lines which contain "ERROR" log level.

And also it skips a line from the total line i.e. if number of lines are 8, it will parese only 7.

Below s my log file:

2016-02-19 11:49:53 ERROR HConnectionManager$HConnectionImplementation:923 - Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
2016-02-19 11:49:53 INFO  HConnectionManager$HConnectionImplementation:1674 - getMaster attempt 1 of 35 failed; retrying after sleep of 100, exception=org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
2016-02-19 11:49:54 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

NOTE: I am using Logstash 2.1.0-all-plugins

2

Your grok expression has two spaces after %{WORD:log_level} but there's only one space between ERROR and HConnectionManager. Change the expression to allow one or more spaces instead of exactly one space.

Don't make a habit out of having multiple GREEDYDATA and DATA patterns in the same expression. It's inefficient and it can backfire.

4

Okay. Thanks a lot @magnusbaeck

5

As a replacement for WORD, you could use the built-in LOGLEVEL pattern instead?

As per - https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns