Last line of logs is not reflected in Logstash and Kibana

[shubhimu (Himanshu)]

Hi,

I am reading log4j logs using filebeat as a service. Everything is working fine for me but last line of logs is not shown in logstash and kibana.

Can someone please tell me what should i do so that last line should be readout properly?

Please see below for my logformat, logstash and filebeat configuration details:

log format:

2018-09-20 15:45:03,003 INFO [Log] [Pass] Deal Button clicked
2018-09-20 15:45:15,386 INFO [Log] [Fail] Results filtered based on Internet and TV

Logstash Configuration:

input{
beats{
port=>5044
host=>"localhost"
}}

filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
mutate {
gsub => ["message", "\n", ""]
}
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
grok {
match => [ "message", "(?m)(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}),%{NUMBER:bytes} %{LOGLEVEL:level} %{SPACE}[%{WORD:logtype}] %{SPACE}[%{WORD:Status}] %{GREEDYDATA:CompleteMessage} "]
}
if "_grokparsefailure" in [tags] {
drop { }
}

}

output {

Print each event to stdout, useful for debugging. Should be commented out in production.

Enabling 'rubydebug' codec on the stdout output will make logstash

pretty-print the entire event as something similar to a JSON representation.

stdout {
codec => rubydebug
}

Sending properly parsed log events to elasticsearch

elasticsearch {
hosts => "localhost:9200"
}
}

Filebeat Configuration :

filebeat.inputs:

• type: log
  enabled: true
  paths:
  • c:\programfiles\elasticsearch\logs*
    document_type: message
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after
    multiline.timeout: 0
    #----------------------------- Logstash output ---------------------------------
    output.logstash:

  Boolean flag to enable or disable the output module.

  enabled: true

  The Logstash hosts

  hosts: ["localhost:5044"]