We have this small setup of ELK Cluster in Production.
ES Nodes : 4 - 3 Master/Data Eligible nodes - 1 Coordinating node Logstash : 3 - Each running on 3 Master/Data Eligible nodes - LoadBalancer Enabled - Performs load balance to our 3 ES servers - Persistent Queus Enabled - queue.type: persisted - queue.max_events: 1000 - queue.max_bytes: 8gb Kibana : 1 - Running on coordinating node machine. Filebeat : 5/19 installed from different servers - multiline_max_lines : 1000
We noticed that, whenever we add few more filebeat source, few events are coming in, sometimes there no events after all from other sources.
For example, we have a new beat source with 50k events to load in the cluster. The other sources goes to 10, a little higher or lower than that.
Then right after maybe 2 and half hours, its stable. Events are coming in smoothly. What might be the issue? Any ideas?