_grokparsefailure

[Jurilz]

Good day. I'm trying to parse tomcat logs with logstash using the filebeat input plugin.

My log-files look like:

2020-11-19 11:34:40,260  [thread0-exec-1] WARN  org.springframework.web.servlet.PageNotFound - Request method 'GET' not supported

This is my logstash.conf:

input {
  beats {
    port => 5144
#    type => "tomcat"
  }
}

filter {
  grok {
    patterns_dir => ["./patterns"]
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{THREAD:thread} %{LOGLEVEL:level} %{JAVALOGMESSAGE:message}" }
  }

  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss, Z" ]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

The 'patterns'-directory contains the file 'thread' with the content 'THREAD \[.*\]'.

This is a part of my output:

}
    },
    "@timestamp" => 2020-11-19T13:40:09.978Z,
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ],
           "log" => {
        "offset" => 11417190,
          "file" => {
            "path" => "/var/log/tomcat8/catalina.out"
}
..<some agent info>..
    },
       "message" => "2020-11-19 14:40:03,624 [thread0-exec-1] WARN  org.springframework.web.servlet.PageNotFound - Request method 'GET' not supported"
}

As you can see i get a _grokparsefailure, although I testet my pattern with 'http://grokconstructor.appspot.com' and it matched.

Unfortunately I do not know where to start here, so any help would be much appreciated. Thank you.

[ylasri (Yassine LASRI)]

try this pattern, it may help

%{TIMESTAMP_ISO8601:timestamp}\s*\[%{DATA:thread}\]\s*%{LOGLEVEL:loglevel}\s*%{JAVACLASS:logclass}\s*%{JAVALOGMESSAGE:message}

You can use Kibana grok debugger to test your patterns quickly

[Jurilz]

Thank you very much. Your solution did the trick.

By the way, I also had to change my date match due to a '_dateparsefailure' to:

filter {
  ...
  date {
    match => [ "timestamp" , "YYYY-MM-dd HH:mm:ss,SSS" ]
  }
}

Thank you again @ylasri