Data missed in Logstash!

[zhaochl]

Data missed a lot in logstash ,version 5.0, is it a serous bug ?

[Christian_Dahlqvist]

You will need to give more information about your configuration in order for anyone to be able to help you. How have you identified that there is data missing? What type of inputs are you using? Which version of Logstash are you using?

[zhaochl]

loogstash version 5.0
I use kibana query log and view by hour, because of my log is rotate by hour, I compare the log line number by linux wc -l command to the kibana count , find that logs in kibana are missed, in all hour log files the last line of the file never read, and in some our files , kibana is mush less than the wc number of the orgginal logs.
my logstash config is as follows:

input {
file {
path => ["/home/work/logs/stat.log*"]
sincedb_path => "/home/work/logstash/bin/db/sincedb-access"
codec=> multiline {
pattern => "^["
negate => true
what => "previous"
}

}
#stdin { }
}

output:
output {
elasticsearch {
hosts => ["172.17.0.33:8081","172.17.0.32:8081"]
index => "stat-%{+YYYY.MM.dd}"
}
#stdout { codec => rubydebug }

}

thanks for any suggestion.

[Christian_Dahlqvist]

If the auto_flush_interval parameter is not set for the multiline codec, the last line will never be released as there is no next line to trigger release on.

[zhaochl]

Yes It woks, thanks a lot
but the log still lost a few ,is it normal for logstash lost several log when transporting the data

thanks a lot