I am re opening this subject because it is also still an issue in alpha5.
Using filebeat witb Kafka. Very happy with performance.
Kafka disk filled up and Kafka stopped ack of lines received.
Filebeats continued to open multiple prospectors until limit of 1024 was reached. Then errors.
When Kafka was rebuilt with more disk capacity it began to receive many duplicate lines. As much as 2.4 million duplicates of one particular line. After many hours still receiving millions of duplicates. 40 million line in Kafka.
Had to add tail_file parameters to filebeats config and delete filebeats registry to recover.
We have about 1000 AWS instances feeding about 6 million lines per day into Kafka.
This is a new environment, yes it is production.
Great work on Beats! We are liking what we see!
Let me know if I can help more.