Last line of logs is not reflected in Logstash and Kibana

(Himanshu) #1


I am reading log4j logs using filebeat as a service. Everything is working fine for me but last line of logs is not shown in logstash and kibana.

Can someone please tell me what should i do so that last line should be readout properly?

Please see below for my logformat, logstash and filebeat configuration details:

log format:

2018-09-20 15:45:03,003 INFO [Log] [Pass] Deal Button clicked
2018-09-20 15:45:15,386 INFO [Log] [Fail] Results filtered based on Internet and TV

Logstash Configuration:


filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
mutate {
gsub => ["message", "\n", ""]
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
grok {
match => [ "message", "(?m)(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}),%{NUMBER:bytes} %{LOGLEVEL:level} %{SPACE}[%{WORD:logtype}] %{SPACE}[%{WORD:Status}] %{GREEDYDATA:CompleteMessage} "]
if "_grokparsefailure" in [tags] {
drop { }


output {

Print each event to stdout, useful for debugging. Should be commented out in production.

Enabling 'rubydebug' codec on the stdout output will make logstash

pretty-print the entire event as something similar to a JSON representation.

stdout {
codec => rubydebug

Sending properly parsed log events to elasticsearch

elasticsearch {
hosts => "localhost:9200"

Filebeat Configuration :


  • type: log
    enabled: true
    • c:\programfiles\elasticsearch\logs*
      document_type: message
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      multiline.timeout: 0
      #----------------------------- Logstash output ---------------------------------

    Boolean flag to enable or disable the output module.

    enabled: true

    The Logstash hosts

    hosts: ["localhost:5044"]

(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.