I think my logstash eating the logs, I am not sure, where I am making the mistake. I am using logstash 2.4, elasticsearch 2.4.1, kibana 4.6.1.
I am not an expert in ELK, but started to explore it. Many times I noticed and felt that logstash losing the logs. so I made a test setup to confirm and test this.
I have a elasticsearch server which is 8 core, 64GB ram, 3.2TB HDD in RAID (15k rpm) running on Centos 7 64bit and Kibana also running in the same server.
I have a seperate server in the same subnet, and connected in the same switch. This server is only for logstash - 16 core, 28gb ram, 600gb hdd. (HP DL 580 G4) running Centos 7 64bit.
Now from my windows workstation I used kiwi syslog message generator to generate 500 syslog messages towards logstash server. Done.
when I look the kibana I can see only 325 logs, I did the testing couple of times, everytime I can see 280 or 325 or 340 or 375, Its varying between 250 to less than 400.
I did a tcpdump during the test, I can see all the 500 logs arrived to the server perfectly. So there is no way for network related issue or packet loss during the transit.
I am not sure what I am missing, Can anyone shed some lights on this?