My logs are missing in elasticsearch and kibana

[irobot678 (meNotHuman)]

Hi ,

my log file had exactly 1,573,249 lines/logs/documents, when i run logstash i can see only 1,541,787 documents in kibana . Around 31k logs/documents are missing and this is too bad .

Can anyone have any solution to this ?
please help me solve this

logstash.conf

input {
file {
path => "/home/Desktop/a.log"
start_position => "beginning"
}
}
filter {

if [message] =~ /\d{15}/ {

grok {
match => ["message","%{TIMESTAMP_ISO8601:date}*[%{LOGLEVEL:log-level}]*%{DATA:thread}*%{DATA:class}*%{DATA:method}*%{DATA:imei}*%{DATA:token}*%{GREEDYDATA:messagedata}"]

}
}

else {

grok {
match => ["message","%{TIMESTAMP_ISO8601:date}*[%{LOGLEVEL:log-level}]*%{DATA:thread}*%{DATA:class}*%{DATA:method}*%{GREEDYDATA:messagedata}"]
add_field => {
"imei" => " "
"token" => " "
}
}
}

}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

and my indices stats show the following :

open logstash-2019.06.27 gkMwatqXRkyML5lpC_DkUA 5 1 1541787 0 248.6mb 248.6mb
status of the index is yellow

can please some one tell me why the logs are missing and solution to over come that.

[zebu14 (xavier)]

Hello,

I would check if my log file doesn't suffer any parse failure during the logstash process.
This could lead to missing documents I think.

[Badger]

Are there any mapping exceptions in the logstash log?

[irobot678 (meNotHuman)]

thanks @Badger and @zebu14 for the reply.

i have added since_db into input of logstash and it worked.