I've been running an ELK stack on a single node for the past year to ingest our application server logs. The load is consistently quite high on this bare-metal box. I was recently given a smaller baremetal server to install an additional instance of logstash on in an attempt to reduce the load on the single node.
I set up logstash with the exact same config on the new server and configured filebeat on the application servers just to send logs to the new box(with just logstash on it). Upon restarting filebeat to use these instances, I noticed that elasticsearch was only ingesting about 10% of the messages it had before. I have no idea where the other 90% went. I thought that perhaps the new box was too slow so maybe the messages were being queued. I then configured filebeat on all the application servers to use load balancing. I encountered the same result as above.
Can someone point me in the right direction to accurately diagnose and fix this?