Data duplication happening when using multiple configuration for logstash with different input ports

Elastic Stack Logstash
#1

Hello,

We have two logstash configurations with two different input ports 5044 and 5045 for accepting connections, but while the filebeat is sending logs to logstash via port 5045 the data is getting duplicated and we are receiving two copies of the actual data.

Is this an expected behavior, if so then is there any option to mention multiple input ports by using the beats input option.

#2

Hey Akhil !

Can you send the filebeat output section from your filebeat.yml ? I hope you are using the in built load-balancer feature of filebeat.

Regards,
Shubham

#3

Currently pointing to a single node out of 3 node cluster of logstash. The moment we start using load balance feature in the output section, the ingestion rate falls down. This again is another issue that's persisting.