Logstash not filter but pattern works

[Akosih]

Hi,

My first topic i hope i will not make a duplicate

I got a problem

I custom my logstash filter to ingest catalina.out logs, the pattern included doesn't fit me so i build this:

Exemple of log:

2017-09-13 12:01:20 [f05e3812] info [native] Command line is /usr/java/default/bin/java -Djava.util.logging.config.file=/opt/tomcat-servers/TOMCAT/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx256m -Xms256m -server -Dorg.apache.catalina.security.SecurityListener.UMASK=0007 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=600 -XX:+UseG1GC -XX:+UseStringDeduplication -DLOG_ROOT_PATH=/opt/applogs/TOMCAT -Dcom.decathlon.environment=PRE_PRODUCTION -Djava.security.egd=file:///dev/urandom -Dinstance.tomcat=TOMCAT -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=rtdkz1rfi01.dktetrix.net --Dmqjazzconfig=/opt/mqjazz/mqmapp/config/flux.properties -Dflows=true -Djava.security.egd=file:///dev/urandom -XX:MaxMetaspaceSize=128M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/applogs/RFID_API01 -XX:OnOutOfMemoryError=/opt/tomcat-servers/bin/manageHPROF.sh_%p -Djdk.tls.ephemeralDHKeySize=2048 -Djava.endorsed.dirs=/opt/apache-tomcat/endorsed -classpath /opt/apache-tomcat/bin/bootstrap.jar:/opt/apache-tomcat/bin/tomcat-juli.jar

my filter

20%{YEAR:Year}-%{MONTHNUM:Month}-%{MONTHDAY:Day} %{TIME:Time} \[%{GREEDYDATA:sequence}\] %{LOGLEVEL:Loglevel} %{GREEDYDATA:error_message}

The pattern work successful

BUT !!!

When the logstash doesnt apply the filter

{
"@timestamp" => 2017-09-14T09:36:43.281Z,
"offset" => 2047,
"@version" => "1",
"input_type" => "log",
"beat" => {
"hostname" => "rtdkz1rfi01.dktetrix.net",
"name" => "rtdkz1rfi01.dktetrix.net",
"version" => "5.5.2"
},
"host" => "rtdkz1rfi01.dktetrix.net",
"source" => "/opt/tomcat-servers/RFID_API01/logs/catalina.out",
"message" => "2017-09-13 12:08:35 [554fc844] info [native] Hot Sensor Placement ........ available",
"type" => "tomcat_catalina_out",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}

Here is my beat configuration (i got multine for information)

• input_type: log
  paths:
  • /opt/tomcat-servers/RFID_API01/logs/catalina.out
    document_type: tomcat_catalina_out
    exclude_lines: ["^---------","DEBUG"]
    include_lines: ["^20"]

And my logstash filter:

  if [type] == "tomcat_catalina_out" {
grok {
  match => { "message" => "20%{YEAR:Year}-%{MONTHNUM:Month}-%{MONTHDAY:Day} %{TIME:Time} %{GREEDYDATA:error_message}" }
     }

}

Do you have any clue ??
Thanks a lot