# My logs are missing in elasticsearch and kibana

**URL:** https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865
**Category:** Logstash
**Created:** [June 27, 2019, 2:16pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865 "2019-06-27T14:16:05Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![irobot678](https://avatars.discourse-cdn.com/v4/letter/i/54ee81/32.png) [@irobot678](https://discuss.elastic.co/u/irobot678)
#### Post date: [June 27, 2019, 2:16pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865/1 "2019-06-27T14:16:05Z")

</div>

Hi ,

my log file had exactly **1,573,249** lines/logs/documents, when i run logstash i can see only **1,541,787** documents in kibana . Around **31k logs/documents are missing** and this is too bad .

Can anyone have any solution to this ?  
please help me solve this

logstash.conf

> input {  
> file {  
> path =\> "/home/Desktop/a.log"  
> start\_position =\> "beginning"  
> }  
> }  
> filter {
> 
> if [message] =~ /\d{15}/ {
> 
> grok {  
> match =\> ["message","%{TIMESTAMP\_ISO8601:date}\*[%{LOGLEVEL:log-level}]\*%{DATA:thread}\*%{DATA:class}\*%{DATA:method}\*%{DATA:imei}\*%{DATA:token}\*%{GREEDYDATA:messagedata}"]
> 
> }  
> }
> 
> else {
> 
> grok {  
> match =\> ["message","%{TIMESTAMP\_ISO8601:date}\*[%{LOGLEVEL:log-level}]\*%{DATA:thread}\*%{DATA:class}\*%{DATA:method}\*%{GREEDYDATA:messagedata}"]   
> add\_field =\> {  
> "imei" =\> " "  
> "token" =\> " "  
> }  
> }  
> }
> 
> }  
> output {  
> elasticsearch { hosts =\> ["localhost:9200"] }  
> stdout { codec =\> rubydebug }  
> }

and my indices stats show the following :

> open logstash-2019.06.27 gkMwatqXRkyML5lpC\_DkUA 5 1 1541787 0 248.6mb 248.6mb  
> status of the index is yellow

can please some one tell me why the logs are missing and solution to over come that.

---

<div class="post-metadata">

### Author: ![zebu14](https://avatars.discourse-cdn.com/v4/letter/z/aca169/32.png) [@zebu14](https://discuss.elastic.co/u/zebu14)
#### Post date: [June 27, 2019, 2:23pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865/2 "2019-06-27T14:23:49Z")

</div>

Hello,

I would check if my log file doesn't suffer any parse failure during the logstash process.  
This could lead to missing documents I think.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 27, 2019, 2:39pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865/3 "2019-06-27T14:39:25Z")

</div>

Are there any mapping exceptions in the logstash log?

---

<div class="post-metadata">

### Author: ![irobot678](https://avatars.discourse-cdn.com/v4/letter/i/54ee81/32.png) [@irobot678](https://discuss.elastic.co/u/irobot678)
#### Post date: [June 28, 2019, 4:18pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865/4 "2019-06-28T16:18:26Z")

</div>

thanks @Badger and @zebu14 for the reply.

i have added since\_db into input of logstash and it worked.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 26, 2019, 4:18pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865/5 "2019-07-26T16:18:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.