# Filebeat 5.0.0-alpha5 multiple harvesters for same file sending the same messages over and over **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 6, 2016, 10:05am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866 "2016-09-06T10:05:57Z")\ **Posts on this page:** 12\ **Page:** 2
**Author:** ![Tim\_Burt](https://avatars.discourse-cdn.com/v4/letter/t/e68b1a/32.png) [@Tim\_Burt](https://discuss.elastic.co/u/Tim_Burt)\ **Post date:** [September 16, 2016, 12:30am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/21 "2016-09-16T00:30:39Z")
Garbage turns out to be coming from our developers... Coincidental with my trying to roll out the new filebeat. Looking good so far. I will know better tomorrow. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 16, 2016, 7:29am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/22 "2016-09-16T07:29:52Z")
🙂 Keep me posted. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 16, 2016, 7:31am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/23 "2016-09-16T07:31:25Z")
@viveklak @Tim_Burt A big thank you to you two for keep investigating this issue and pushing forward to get it fixed. I really appreciate all the work you put into this one. ---
**Author:** ![Tim\_Burt](https://avatars.discourse-cdn.com/v4/letter/t/e68b1a/32.png) [@Tim\_Burt](https://discuss.elastic.co/u/Tim_Burt)\ **Post date:** [September 17, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/24 "2016-09-17T13:13:15Z")
Just got another one.. But perhaps more useful info. [root@ip-172-16-59-236 rails]# filebeat -version filebeat version 6.0.0-alpha1 (amd64), libbeat 6.0.0-alpha1 ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c04945de6f587a3668b4d5c6755e9f340f81c591.png) Looks like 6 million identical records in 2 ms. Here is the caveat.. I have a developer that is dumping a large JSON object into the rails log files. This single line is approx 118 K in length. These lines are prevalent in the rails logs The line that is duplicated in the Kibana screenshot is surrounded by these long lines. Could it be possible this could be a buffer overrun? I can share with you the raw logs but not in this forum. You can contact me at: removed ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 19, 2016, 7:42am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/25 "2016-09-19T07:42:39Z")
@Tim_Burt Just sent you an email, interested to see the log file. Can we reopen a new thread for this topic? I get the impression it is also an issue but not directly related to the previous one we that got fixed. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 21, 2016, 7:06am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/26 "2016-09-21T07:06:03Z")
Thanks for the log file. I tried to reproduce the issue locally but couldn't so far. I tested on my side with File output instead of Kafka to take Kafka out of the equation. Do you see this happen every time such a JSON thingy is in the log or only from time to time? Can you reproduce it locally on your side with Kafka? One idea could be that Kafka only accepts a certain message size? ---
**Author:** ![Tim\_Burt](https://avatars.discourse-cdn.com/v4/letter/t/e68b1a/32.png) [@Tim\_Burt](https://discuss.elastic.co/u/Tim_Burt)\ **Post date:** [September 22, 2016, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/27 "2016-09-22T15:03:55Z")
Thanks for the effort... Yes, I am aware that my issues may be with Kafka, it is difficult to discern which element is misbehaving. I do have a test environment for file input, but I have not had the time at the moment. We just did a migration and we are all still scrambling with the mop-up. Your fix seems to have done the trick. I am no longer getting the millions of duplicates. i have seen 2 or 4 duplicates of a line, but that may have a different origin entirely. I will need to do much more research. Thank you for all of your help and hard work. We are looking forward to 5.0.0. I think we can close this topic for now. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 23, 2016, 6:39am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/28 "2016-09-23T06:39:15Z")
Please ping us when you hit issue again (also with Kafka 😉 ). Side note: Beta1 was just released: [https://www.elastic.co/blog/elastic-stack-release-5-0-0-beta1](https://www.elastic.co/blog/elastic-stack-release-5-0-0-beta1) ---
**Author:** ![Tim\_Burt](https://avatars.discourse-cdn.com/v4/letter/t/e68b1a/32.png) [@Tim\_Burt](https://discuss.elastic.co/u/Tim_Burt)\ **Post date:** [September 26, 2016, 11:40pm UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/29 "2016-09-26T23:40:54Z")
Ping.... I sent you an email with logs and configs.... Nothing I cannot handle, but it may help you solve a corner case. Thanks again for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 27, 2016, 10:06am UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/30 "2016-09-27T10:06:08Z")
This topic was automatically closed after 21 days. New replies are no longer allowed. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 2, 2016, 7:54pm UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/31 "2016-10-02T19:54:58Z")
I will look into it as soon as I get the time and will get back to you. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 10, 2016, 12:02pm UTC](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866/32 "2016-10-10T12:02:27Z")
For everyone going through this Thread, we found the potential problem here: [Filebeat 5.0beta1 with kafka output: multiplication of log lines](https://discuss.elastic.co/t/filebeat-5-0beta1-with-kafka-output-multiplication-of-log-lines/62513) [Previous page](https://discuss.elastic.co/t/filebeat-5-0-0-alpha5-multiple-harvesters-for-same-file-sending-the-same-messages-over-and-over/59866.md?page=1)