# Filebeat sends duplicate log

**URL:** https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926
**Category:** Beats
**Tags:** filebeat
**Created:** [October 23, 2019, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926 "2019-10-23T17:31:56Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Pepper\_Pam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pepper_pam/32/56455_2.png) [@Pepper\_Pam](https://discuss.elastic.co/u/Pepper_Pam)
#### Post date: [October 23, 2019, 5:31pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926/1 "2019-10-23T17:31:57Z")

</div>

I used filebeat to connect Kafka, and I noticed that everytime I updated the log file, the filebeat will send duplicate content to the Kafka. Any help to change that?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 23, 2019, 5:41pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926/2 "2019-10-23T17:41:59Z")

</div>

Exactly how did you update the file? If you used an editor a new file is generally created behind the scenes and replaced, which could be why it is reread for every modification. Filebeat uses inodes to keep track of files and not just the name and path. Instead make sure you append to the file, e.g. using a script or `echo "data" >> testfile.log`.

---

<div class="post-metadata">

### Author: ![Pepper\_Pam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pepper_pam/32/56455_2.png) [@Pepper\_Pam](https://discuss.elastic.co/u/Pepper_Pam)
#### Post date: [October 23, 2019, 6:06pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926/3 "2019-10-23T18:06:42Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> `echo "data" >> testfile.log` .

This is really helpful.  
After I used the script like you mention:echo "data" \>\> testfile.log  
I can only read the new add lines from the filebeat.

But I still got duplicate log. That means the filebeat output the same content twice.

 ![43%20PM](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26ab68cb54e83f4156eb3f0ffcbd1210d9246199.png)

---

<div class="post-metadata">

### Author: ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)
#### Post date: [October 23, 2019, 11:44pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926/4 "2019-10-23T23:44:08Z")

</div>

Can you please check the filebeat log ? Filebeat publishes metrics in the log files and it could give you a clue.  
Default location : /var/log/filebeat/filebeat

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 20, 2019, 11:44pm UTC](https://discuss.elastic.co/t/filebeat-sends-duplicate-log/204926/5 "2019-11-20T23:44:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.