# Moving from Logstash to Filebeat =\> no duplicate log

**URL:** <https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 7, 2016, 10:33am UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278 "2016-12-07T10:33:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nicolas\_Guyomar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_guyomar/32/10124_2.png) [@Nicolas\_Guyomar](https://discuss.elastic.co/u/Nicolas_Guyomar)\
**Post date:** [December 7, 2016, 10:33am UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278/1 "2016-12-07T10:33:14Z")

</div>

Hi everyone,

I would like to move from logstash as a log shipper, to filebeat.

I'm using the logstash file input plugin to collect logs, same thing with filebeat, to send everything to a centralized logstash-shipper before writing to elasticsearch.

The thing is, if I shutdown logstash and start a fresh filebeat instance instead, filebeat will start from the beginning of the file, leading to duplicate logs in Elasticsearch.

I could have add a "log content hash based" on logstash-shipper side in elasticsearch document\_id to avoid duplicates, but I have to admit I'd like an easier solution.

Would you have any idea on how to "bootstrap" a filebeat instance with logstash file cursor maybe ?

Thank you

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [December 7, 2016, 11:50am UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278/2 "2016-12-07T11:50:57Z")

</div>

I never tried it but I think it should be possible to write a small script in your preferred language that takes the sincedb from LS and converts it into a filebeat registry file. An alternative is using `tail_files` in filebeat, but if during shutdown LS and boot up Filebeat log lines were added, these are lost.

Other solution could be, that you write filebeat logs to a different index and then manually check (based on the timestamp?) what the time range of the duplicated events is and then use delete\_by\_query to remove these from of the two indices. That would mean running both for a certain. This is also what I would recommend.

---

<div class="post-metadata">

**Author:** ![Nicolas\_Guyomar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_guyomar/32/10124_2.png) [@Nicolas\_Guyomar](https://discuss.elastic.co/u/Nicolas_Guyomar)\
**Post date:** [December 7, 2016, 1:02pm UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278/3 "2016-12-07T13:02:14Z")

</div>

Hi ruflin,

Thanks for such a quick answer !

Maybe loosing some logs using tail\_files will be acceptable for my client.  
Converting the sincedb into a filebeat registry file seems ok to me, I'll look into it.

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2017, 1:02pm UTC](https://discuss.elastic.co/t/moving-from-logstash-to-filebeat-no-duplicate-log/68278/4 "2017-01-04T13:02:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.