# Filebeat sending the whole log again

**URL:** <https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 23, 2019, 1:58pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886 "2019-10-23T13:58:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeto.abialinti](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@jeto.abialinti](https://discuss.elastic.co/u/jeto.abialinti)\
**Post date:** [October 23, 2019, 1:58pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886/1 "2019-10-23T13:58:39Z")

</div>

I encounter a bug on filebeat, Filebeat sending the whole log again  
Is there a way for filebeat not to read log history so far, but rather for the last 60 to 100 lines?

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [October 23, 2019, 5:08pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886/2 "2019-10-23T17:08:09Z")

</div>

Filebeat uses registry to track what files it read and for each file the offset.  
It is not designed to start reading from any random location.

---

<div class="post-metadata">

**Author:** ![jeto.abialinti](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@jeto.abialinti](https://discuss.elastic.co/u/jeto.abialinti)\
**Post date:** [October 23, 2019, 5:35pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886/3 "2019-10-23T17:35:17Z")

</div>

Is there a way to force filebeat not to sending again log lines that have already sent

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [October 23, 2019, 6:03pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886/4 "2019-10-23T18:03:14Z")

</div>

You mean to not send duplicate lines if the same lines have been sent earlier.  
It is not possible directly in Filebeat but you can go through [https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) to remove duplicates on ElasticSearch side

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2019, 6:03pm UTC](https://discuss.elastic.co/t/filebeat-sending-the-whole-log-again/204886/5 "2019-11-20T18:03:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.