Grokparse failures issues

Elastic Stack Logstash
1

We have created a logstash configuration with tomcat logs as the input and provide as the output in Json format to a file. This is the configuration :

===================================================================
input {
  file {
    path => "/opt/apache-tomcat-8.5.11/logs/catalina.out"
    type => "tomcat-logs"
    start_position => "beginning"
  }
}

filter {
  grok  {
         match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:message}" }
        }
       }

output {
        file
       {
         codec => rubydebug
         path => "/root/logstashoutput/filtered-logs_second1.txt"
       }
}

====================================================================

But there are some parse failures and thus _grokparsefailure written to the output file.
I have created settings to remove the _grokparsefailure using the following configuration :slight_smile:

============================================================

input {
file {
path => "/root/tomcat-logs.txt"
type => "tomcat-logs"
start_position => "beginning"
}
}

filter {
  grok  {
         match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:message}" }
        }
     }

output {
     if ! ("_grokparsefailure" in [tags])
     {
       file
       {
         codec => rubydebug
         path => "/root/logstashoutput/Excluded_parse_failure.txt"
       }
 }
}

============================================================

But we are getting the output as follows :

=============================================

"host" => "Hostname",
"@version" => "1",
"message" => [
[0] "2018-10-12 01:00:00.763 DEBUG 30434 --- message",
[1] "30434 --- [pool-2-thread-1] message",
[2] "30434 --- [pool-2-thread-1] message",
[3] "30434 --- [pool-2-thread-1] message",
[4] "30434 --- [pool-2-thread-1] message"
],
"@timestamp" => 2018-10-11T19:30:01.644Z,
"type" => "tomcat-logs",
"path" => "/opt/apache-tomcat-8.5.11/logs/catalina.out",
"timestamp" => [
[0] "2018-10-12 01:00:00.763",
[1] "2018-10-12 01:00:00.763",
[2] "2018-10-12 01:00:00.763",
[3] "2018-10-12 01:00:00.763"
],
"log_level" => [
[0] "DEBUG",
[1] "DEBUG",
[2] "DEBUG",
[3] "DEBUG"
]
}
=======================================================

The message,log_level, timestamp,type,path etc got mixed. 

I want the output as :      

--------------------------------------------------------------
{
      "host" => "Hostname",
      "path" => "/opt/apache-tomcat-8.5.11/logs/catalina.out",
   "message" => [
    [0] "<message from the logs>"
],
      "type" => "tomcat-logs",
 "timestamp" => "2018-10-11 14:31:49.679",
  "@version" => "1",
 "log_level" => "DEBUG"
}
-----------------------------------------------------------------

Tried 

------------------
 if "_grokparsefailure" not in [tags]
{
  file
   {
     codec => rubydebug
     path => "/root/logstashoutput/Directly_from_tomcat.txt"
   }
   }
 }

and

 if "_grokparsefailure"  in [tags]{
           drop { }
     }
     }

But same results.

May I know what are the changes required to be made to the configuration.

Awaiting your reply.

2

You are overwriting message with message in your GREEDYDATA filter.
Don't do that :stuck_out_tongue:
Change:
%{GREEDYDATA:message}
to
%{GREEDYDATA:log_message}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.