# Data loss frequently

**URL:** <https://discuss.elastic.co/t/data-loss-frequently/147201>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 4, 2018, 12:17pm UTC](https://discuss.elastic.co/t/data-loss-frequently/147201 "2018-09-04T12:17:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![manish\_jaiswal](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@manish\_jaiswal](https://discuss.elastic.co/u/manish_jaiswal)\
**Post date:** [September 4, 2018, 12:17pm UTC](https://discuss.elastic.co/t/data-loss-frequently/147201/1 "2018-09-04T12:17:05Z")

</div>

we have more than 135 filebeat running on different instance. each sending data to more than 10 pipeline.

getting error on most of the filebeat(Failed to publish events: temporary bulk send failure):

2018-09-04T17:39:22.326+0530 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.3.1  
2018-09-04T17:39:22.331+0530 INFO template/load.go:73 Template already exists and will not be overwritten.  
2018-09-04T17:39:22.331+0530 INFO [publish] pipeline/retry.go:172 retryer: send unwait-signal to consumer  
2018-09-04T17:39:22.331+0530 INFO [publish] pipeline/retry.go:174 done  
2018-09-04T17:39:22.341+0530 INFO [publish] pipeline/retry.go:149 retryer: send wait signal to consumer  
2018-09-04T17:39:22.341+0530 INFO [publish] pipeline/retry.go:151 done  
2018-09-04T17:39:22.973+0530 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150580,"time":{"ms":3}},"total":{"ticks":1106600,"time":{"ms":75},"value":1106600},"user":{"ticks":956020,"time":{"ms":72}}},"info":{"ephemeral\_id":"c0c77725-d7ec-4d04-9778-6c3e87caf483","uptime":{"ms":271440046}},"memstats":{"gc\_next":20433952,"memory\_alloc":18759592,"memory\_total":81088606696}},"filebeat":{"harvester":{"open\_files":10,"running":10}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":29,"failed":89,"total":89},"read":{"bytes":30628},"write":{"bytes":53426}},"pipeline":{"clients":96,"events":{"active":4126,"retry":178}}},"registrar":{"states":{"current":11}},"system":{"load":{"1":0.73,"15":0.21,"5":0.27,"norm":{"1":0.0913,"15":0.0263,"5":0.0338}}},"xpack":{"monitoring":{"pipeline":{"events":{"published":3,"total":3},"queue":{"acked":3}}}}}}}  
2018-09-04T17:39:23.341+0530 ERROR pipeline/output.go:92 Failed to publish events: temporary bulk send failure  
2018-09-04T17:39:23.341+0530 INFO [publish] pipeline/retry.go:172 retryer: send unwait-signal to consumer  
2018-09-04T17:39:23.341+0530 INFO [publish] pipeline/retry.go:174 done  
2018-09-04T17:39:23.341+0530 INFO [publish] pipeline/retry.go:149 retryer: send wait signal to consumer  
2018-09-04T17:39:23.341+0530 INFO [publish] pipeline/retry.go:151 done  
2018-09-04T17:39:23.342+0530 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.3.1  
2018-09-04T17:39:23.344+0530 INFO template/load.go:73 Template already exists and will not be overwritten.  
2018-09-04T17:39:23.344+0530 INFO [publish] pipeline/retry.go:172 retryer: send unwait-signal to consumer  
2018-09-04T17:39:23.344+0530 INFO [publish] pipeline/retry.go:174 done  
2018-09-04T17:39:23.346+0530 INFO [publish] pipeline/retry.go:149 retryer: send wait signal to consumer  
2018-09-04T17:39:23.346+0530 INFO [publish] pipeline/retry.go:151 done

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 4, 2018, 4:30pm UTC](https://discuss.elastic.co/t/data-loss-frequently/147201/2 "2018-09-04T16:30:12Z")

</div>

It seems like Elasticsearch cannot handle the load. What do you see in logs of ES?

To minimize data loss, you could use spooling to disk. It's available since 6.3. [https://www.elastic.co/guide/en/beats/filebeat/6.3/configuring-internal-queue.html#configuration-internal-queue-spool](https://www.elastic.co/guide/en/beats/filebeat/6.3/configuring-internal-queue.html#configuration-internal-queue-spool)  
It stores the messages in a file until they can be forwarded to the output.  
You could also increase the size of the mem queue. But obviously, it uses memory. So you could opt for spooling to disk.

---

<div class="post-metadata">

**Author:** ![manish\_jaiswal](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@manish\_jaiswal](https://discuss.elastic.co/u/manish_jaiswal)\
**Post date:** [September 5, 2018, 10:48am UTC](https://discuss.elastic.co/t/data-loss-frequently/147201/3 "2018-09-05T10:48:08Z")

</div>

will spooling disk slow down logging in kibana. and it is in beta version will not recommended to use in production

---

<div class="post-metadata">

**Author:** ![manish\_jaiswal](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@manish\_jaiswal](https://discuss.elastic.co/u/manish_jaiswal)\
**Post date:** [September 5, 2018, 10:59am UTC](https://discuss.elastic.co/t/data-loss-frequently/147201/4 "2018-09-05T10:59:25Z")

</div>

i am also getting gc overhead on elastic search cluster on each instance.

instance memory 60  
heap size ES :32  
queue size:7000  
thread pool:write

Please help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2018, 10:59am UTC](https://discuss.elastic.co/t/data-loss-frequently/147201/5 "2018-10-03T10:59:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.