# Some logs are missing in Elasticsearch

**URL:** <https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214>\
**Category:** Logstash\
**Created:** [June 21, 2017, 7:45am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214 "2017-06-21T07:45:54Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 7:45am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/1 "2017-06-21T07:45:55Z")

</div>

Hi,

We have a 3 node cluster ( 2 data nodes and 1 ingest node ) which is Centos 7.3, openjdk version "1.8.0\_121", Logstash 5.2 and Elasticsearch 5.2. We receives the log files every 5 minutes. We found that some logs are missing on the elasticserch sometimes. If we duplicate the file, all logs will be imported to the cluster. We didn't see any error messages from both Logstash and Elasticsearch logs. We have tried to update the Logstash to 5.4.1 and the problem is still exist. We also tried to add a file output on Logstash. The problem is the same on the output file. The total number of imported logs is the same as Elasticsearch. Seems the problem is caused by Logstash. We have no idea on how and when it happen.

## Screen Captures

![](https://us1.discourse-cdn.com/elastic/original/3X/8/3/832c2ca2ff5e39fc4f7492ec815e22baa84799b1.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/8/7/8737a043062381d3e8038f397d8cf954538987ea.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/8/3/83f66914d50b7e75fc85194e90f63e2e032cd7a8.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d723925aab75dbaf590c380094f221fd8456d869.png)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/1/b/1bd8befd5bd7e26ca1452c15c4313c9f0561c7a4.png)

Regards,

Ricky

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 7:55am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/2 "2017-06-21T07:55:27Z")

</div>

Add the configuration for reference

## logstash config file

input {  
file {  
path =\> "/var/log/xxxxxx/\*\_pcap.json"  
sincedb\_path =\> "/var/log/logstash/tshark-sincedb"  
#interval =\> 300  
type =\> "Check\_OK"  
start\_position=\> "beginning"  
}

file {  
path =\> "/var/log/xxxxxx/\*\_err.json"  
sincedb\_path =\> "/var/log/logstash/tshark-err-sincedb"  
#interval =\> 300  
type =\> "Check\_Fail"  
start\_position=\> "beginning"  
}  
}

filter{  
json{  
source =\> "message"  
}

```
fingerprint{
 concatenate_sources => true
 method => "SHA1"
 key => "xxxxxx-elasticsearch"
 source => ["timestamp","Source_IP","Destination_IP","Application_ID","Command_Code","Flags_Request","Session_Id","Origin_Realm","Origin_Host","Destination_Realm","Destination_Host","User_Name","Result_Code","Experimental_Result_Code","RAT_Type","CC_Request_Type","CC_Request_Number","Service_Context_Id","User_Equipment_Info_Value","PDP_Address_IPv4","Rule_Space_Decision","MME_Name","missing"]
}

date {
match => ["timestamp", "MMM dd, YYYY HH:mm:ss.SSSSSSSSS ZZ", "MMM dd, YYYY HH:mm:ss.SSSSSSSSS ZZ"]
}

```

if [type] == "Check\_OK" {  
mutate {  
add\_field =\> {  
"Command\_Name\_temp" =\> "%{Command\_Code}%{Flags\_Request}"  
}  
}

```
translate {
   field => "Application_ID"
   destination => "Application_Name"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/xxxxxx_dictionary.yml"
   }

translate {
   field => "Command_Name_temp"
   destination => "Command_Name"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/xxxxxx_dictionary.yml"
   remove_field => ["Command_Name_temp"]
   }

translate {
   exact => true
   regex => true
   field => "Origin_Realm"
   destination => "Origin_Provider"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/MNC-MCC-PLNM.yaml"
   }

translate {
   exact => true
   regex => true
   field => "Destination_Realm"
   destination => "Destination_Provider"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/MNC-MCC-PLNM.yaml"
   }

translate {
   exact => true
   regex => true
   field => "Origin_Realm"
   destination => "Origin_Country"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/MCC-Country.yaml"
   }

translate {
   exact => true
   regex => true
   field => "Destination_Realm"
   destination => "Destination_Country"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/MCC-Country.yaml"
   }

translate {
   exact => true
   field => "Origin_Country"
   destination => "Origin_Location_temp"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/Country-LatLong.json"
   }

translate {
   exact => true
   field => "Destination_Country"
   destination => "Destination_Location_temp"
   fallback => "Unknown"
   dictionary_path => "/var/log/logstash/Country-LatLong.json"
   }

translate {
   exact => true
   regex => true
   field => "Source_IP"
   destination => "Source_Host_temp"
   fallback => "Unknown,Unknown"
   dictionary_path => "/var/log/logstash/xxxxxx-hosts.yaml"
   }

translate {
   exact => true
   regex => true
   field => "Destination_IP"
   destination => "Destination_Host_temp"
   fallback => "Unknown,Unknown"
   dictionary_path => "/var/log/logstash/xxxxxx-hosts.yaml"
   }

if ("" in [Source_Host_temp]) {
   mutate {
     split => { "Source_Host_temp" => "," }
     add_field => { 
             "Source_Host_Name" => "%{[Source_Host_temp][0]}" 
             }
     add_field => { 
             "Source_Host_Type" => "%{[Source_Host_temp][1]}" 
             }
     remove_field => ["Source_Host_temp"]
       }
 }

if [Source_Host_Type] == "IPX" {
   translate {
     exact => true
     regex => true
     field => "Source_Host_Name"
     destination => "Source_IPX"
     fallback => "Unknown"
     dictionary_path => "/var/log/logstash/xxxxxx-ipx.yaml"
       }
 }

if ("" in [Destination_Host_temp]) {
   mutate {
     split => { "Destination_Host_temp" => "," }
     add_field => { 
             "Destination_Host_Name" => "%{[Destination_Host_temp][0]}" 
             }
     add_field => { 
             "Destination_Host_Type" => "%{[Destination_Host_temp][1]}" 
             }
     remove_field => ["Destination_Host_temp"]
       }
 }

if [Destination_Host_Type] == "IPX" {
   translate {
     exact => true
     regex => true
     field => "Destination_Host_Name"
     destination => "Destination_IPX"
     fallback => "Unknown"
     dictionary_path => "/var/log/logstash/xxxxxx-ipx.yaml"
       }
 }

if ("" in [Origin_Country]) {
   mutate {
     split => { "Origin_Location_temp" => "," }
     add_field => ["[Origin_Location][lat]", "%{[Origin_Location_temp][0]}" ]
     add_field => ["[Origin_Location][lon]", "%{[Origin_Location_temp][1]}" ]
 remove_field => ["Origin_Location_temp"]
       }
 }

if ("" in [Destination_Country]) {
   mutate {
     split => { "Destination_Location_temp" => "," }
     add_field => ["[Destination_Location][lat]", "%{[Destination_Location_temp][0]}" ]
     add_field => ["[Destination_Location][lon]", "%{[Destination_Location_temp][1]}" ]
     remove_field => ["Destination_Location_temp"]
     }
  # remove_field => ["Origin_Location_temp", "Destination_Location_temp"]
  }
   
 }

```

}

output {  
if [type] == "Check\_OK" {  
elasticsearch {  
action =\> "index"  
hosts =\> ["10.192.0.178:9200","10.192.0.147:9200"]  
index =\> "tshark-%{+YYYYMMdd}"  
document\_id =\> "%{fingerprint}"  
flush\_size =\> 5000  
}

```
	file {
		path => "/var/log/xxxxxx/output_test_%{+YYYYMMdd}.txt"
		codec => "json_lines"
		}
       }

      if [type] == "Check_Fail" {
	elasticsearch {
		action => "index"
		hosts => ["10.192.0.178:9200","10.192.0.147:9200"]
		index => "tshark-err-%{+YYYYMMdd}"
		document_id => "%{fingerprint}"
		flush_size => 5000
	}
           file {
                    path => "/var/log/xxxxxx/output_err_test_%{+YYYYMMdd}.txt"
                    codec => "json_lines"
                    }
       }

```

}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 21, 2017, 8:02am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/3 "2017-06-21T08:02:49Z")

</div>

What does `_cat/indices` show? Do you have any deletes from a single load that may indicate hash collisions? How many logs are missing?

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 8:22am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/4 "2017-06-21T08:22:56Z")

</div>

All indices are green and open. Some logs are duplicated between files, around 1600 lines. We configured Logstash to generate the fingerprint by all fields of the event. Those duplicate logs will be overwritten by the new log files. We didn't see any message about hash collisions. When the problem occurs, over 80% logs are missing.

fingerprint{  
concatenate\_sources =\> true  
method =\> "SHA1"  
key =\> "xxxxxx-elasticsearch"  
source =\> ["timestamp","Source\_IP","Destination\_IP","Application\_ID","Command\_Code","Flags\_Request","Session\_Id","Origin\_Realm","Origin\_Host","Destination\_Realm","Destination\_Host","User\_Name","Result\_Code","Experimental\_Result\_Code","RAT\_Type","CC\_Request\_Type","CC\_Request\_Number","Service\_Context\_Id","User\_Equipment\_Info\_Value","PDP\_Address\_IPv4","Rule\_Space\_Decision","MME\_Name","missing"]  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 21, 2017, 8:30am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/5 "2017-06-21T08:30:47Z")

</div>

What does `_cat/indices` show for the index where documents are missing?

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 8:48am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/6 "2017-06-21T08:48:54Z")

</div>

The following is the results from 'wc -l' for the total no. of logs

 ![](https://us1.discourse-cdn.com/elastic/original/3X/7/2/722f9712ab390af244b47707d5cc82c0ff301fd2.png)

\_cat/indices

 ![](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b974f74bab6e1021e324d9315d7a00e2c7d2dda4.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 21, 2017, 8:56am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/7 "2017-06-21T08:56:38Z")

</div>

Please do not post screenshots of text as it is very hard to read. It looks like you have a significant number of duplicates that have resulted in updates (shown as deleted documents). Is it perhaps possible that the fingerprint calculation fails for a portion of records and you end up with a document named `%{fingerprint}` (no variable substitution) being updated for lots of different records? Can you search and see if you have a record with `%{fingerprint}` as an id?

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 9:04am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/8 "2017-06-21T09:04:11Z")

</div>

Those deleted documents are the duplicataed entries between log files. Those logs files are network packet captures. They have 12 seconds overlap time frame. It have 300000 documents duplicated a day.

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 9:13am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/9 "2017-06-21T09:13:28Z")

</div>

I have searched on Kibana, no record with %{fingerprint} as \_id

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 21, 2017, 9:19am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/10 "2017-06-21T09:19:03Z")

</div>

Is there anything in the logs?

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 9:20am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/11 "2017-06-21T09:20:21Z")

</div>

No error on both logstash and elastic search.

---

<div class="post-metadata">

**Author:** ![ricky.chiu](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\
**Post date:** [June 21, 2017, 9:51am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/12 "2017-06-21T09:51:22Z")

</div>

May I know any limitation on total no. of documents? We have 541829108 documents right now. We notice this problem when the cluster reached 300000000 documents. Normally, it have 11000000 documents a day. Seems the more data stored, the problem occurs more often.

![](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7be9e24cd2e6165f1426df03f8ddd2793df1c88c.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2017, 9:51am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214/13 "2017-07-19T09:51:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.