LogStash filter not works with FileBeat input

Elastic Stack Logstash
1

Hi,

I have an ELK stack running on my server. Everything looks good except when I tried to filter out the timestamp from the message part of catalina.out log file.

The below is the message.

2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119

The below is the LogStash configuration.

input {
beats {
port => 5044
}
}

filter {
grok {
match => [ "message", "(?(\d){4}-(\d){2}-(\d){2} (\d){2}:(\d){2}:(\d){2},(\d){3})" ]
}
date {
match => [ "sourcestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
user => "elastic"
password => "*******"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Below the filebeat.yml configuration.

filebeat.prospectors:

  • type: log
    enabled: true
    paths:
    • /var/log/tomcat/catalina.*
      multiline.pattern: '^java|^[[:space:]]+(at)'
      multiline.negate: false
      multiline.match: after
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      setup.template.settings:
      index.number_of_shards: 3
      setup.kibana:
      output.logstash:
      hosts: ["127.0.0.1:5044"]

The log is not coming up in Kibana. I need the log in Kibana with timestamp from the message part. Please help with that. Not getting any idea from the logs. If you need, I can share the logs as well here.

Note: When I tried from the terminal using Logstash command with stdin and stdout(given below), I got the required output. But it is not working with filebeat-logstash combination.

~$ echo "2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor.testcatalina (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119" | /usr/share/logstash/bin/logstash -f conf.d/logstash.conf --path.settings /etc/logstash/

~$ _2018-02-07 17:29:21,704 main ERROR Unable to locate appender "${sys:ls.log.format}rolling" for logger config "root"
2018-02-07 17:29:23,841 main ERROR Unable to locate appender
"${sys:ls.log.format}_rolling" for logger config "root"
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
{
"sourcestamp" => "2018-01-23 12:49:17,944",
"host" => "QB-IPL-1205",
"@version" => "1",
"message" => "2018-01-23 12:49:17,944 [http-nio-8080-exec-6] INFO org.napa.web.rest.interceptor.RequestVersionInterceptor.testcatalina (RequestVersionInterceptor.java:34) - App Version: 2.1.1.3/null App type: by userId: 2119",
"@timestamp" => 2018-01-23T07:19:17.944Z
}

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.