# Es loss applog data

**URL:** <https://discuss.elastic.co/t/es-loss-applog-data/207988>\
**Category:** Elasticsearch\
**Created:** [November 15, 2019, 3:03am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988 "2019-11-15T03:03:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gydtc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gydtc/32/54695_2.png) [@gydtc](https://discuss.elastic.co/u/gydtc)\
**Post date:** [November 15, 2019, 3:03am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/1 "2019-11-15T03:03:53Z")

</div>

my applog is applog -\> kafka -\> logstash -\> es  
but every day，it will loss 10 sec data during 23:59:50 - 00:00:10  
who can help me  
es 7.4  
logstash 7.4

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 15, 2019, 6:08am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/2 "2019-11-15T06:08:08Z")

</div>

Does this period coincide with new daily indices being created? If this is the case the fact that no data is indexed during this period does not mean data is lost as Kafka would buffer it. If creating new daily indices take that long it could however indicate that you have a performance problem in the cluster or that you simply have too many indices and shard, making cluster state updates slow. How many indices are you creating per day? How many shard do you have in the cluster? How large is the cluster?

---

<div class="post-metadata">

**Author:** ![gydtc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gydtc/32/54695_2.png) [@gydtc](https://discuss.elastic.co/u/gydtc)\
**Post date:** [November 15, 2019, 6:27am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/3 "2019-11-15T06:27:46Z")

</div>

thanks。  
this is a new elk system，the indices are less than 30.  
and i will create the daily indices during that time.  
my pipleline is like this:

input {  
kafka {  
bootstrap\_servers =\> "......"  
consumer\_threads =\> "......"  
session\_timeout\_ms =\> "......"  
}  
}  
filter {  
json {  
source =\> "message"  
}  
ruby {  
code =\> "event.set('my\_index\_day', event.timestamp.time.localtime.strftime('%Y.%m.%d'))"  
}  
}  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["......"]  
index =\> "applog-%{my\_index\_day}"  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 15, 2019, 6:55am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/4 "2019-11-15T06:55:29Z")

</div>

What type of storage are you using for Elasticsearch? What is the specification of the Elasticsearch nodes in terms of CPU, RAM, heap and storage?

---

<div class="post-metadata">

**Author:** ![gydtc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gydtc/32/54695_2.png) [@gydtc](https://discuss.elastic.co/u/gydtc)\
**Post date:** [November 16, 2019, 1:39am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/5 "2019-11-16T01:39:11Z")

</div>

just storing apache log。  
my es nodes  
cpu 2socket 16core  
ram 128g  
heapsize 31g  
storage 40Tb

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 16, 2019, 5:47am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/6 "2019-11-16T05:47:55Z")

</div>

What type of storage are you using? Is it locally attached disks? Is there anything in the Elasticsearch logs around that time? How many nodes do you have?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 14, 2019, 5:47am UTC](https://discuss.elastic.co/t/es-loss-applog-data/207988/7 "2019-12-14T05:47:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.