LogStash filter not works with FileBeat input

[Uwais_Ibrahim]

Hi,

I have an ELK stack running on my server. Everything looks good except when I tried to filter out the timestamp from the message part of catalina.out log file.

The below is the message.

2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119

The below is the LogStash configuration.

input {
beats {
port => 5044
}
}

filter {
grok {
match => [ "message", "(?(\d){4}-(\d){2}-(\d){2} (\d){2}:(\d){2}:(\d){2},(\d){3})" ]
}
date {
match => [ "sourcestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
user => "elastic"
password => "*******"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Below the filebeat.yml configuration.

filebeat.prospectors:

• type: log
  enabled: true
  paths:
  • /var/log/tomcat/catalina.*
    multiline.pattern: '^java|^[[:space:]]+(at)'
    multiline.negate: false
    multiline.match: after
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
    setup.template.settings:
    index.number_of_shards: 3
    setup.kibana:
    output.logstash:
    hosts: ["127.0.0.1:5044"]

The log is not coming up in Kibana. I need the log in Kibana with timestamp from the message part. Please help with that. Not getting any idea from the logs. If you need, I can share the logs as well here.

Note: When I tried from the terminal using Logstash command with stdin and stdout(given below), I got the required output. But it is not working with filebeat-logstash combination.

~$ echo "2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor.testcatalina (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119" | /usr/share/logstash/bin/logstash -f conf.d/logstash.conf --path.settings /etc/logstash/

~$ _2018-02-07 17:29:21,704 main ERROR Unable to locate appender "${sys:ls.log.format}rolling" for logger config "root"
2018-02-07 17:29:23,841 main ERROR Unable to locate appender
"${sys:ls.log.format}_rolling" for logger config "root"
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
{
"sourcestamp" => "2018-01-23 12:49:17,944",
"host" => "QB-IPL-1205",
"@version" => "1",
"message" => "2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor.testcatalina (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119",
"@timestamp" => 2018-01-23T07:19:17.944Z
}