We have an ELK cluster with 3 hosts on Ubuntu 14.04 (logstash 2.4.1+ES 2.4.1 on each host in Docker containers), with logstash configured to receive gelf and syslog/tcp logs.
Sometimes some gelf logs are lost.
- With tcdump on the server, I see that the UDP packet reaches the host
netstat -c --udp -an | grep 12201I see the udp queue is always empty
- logstash is using nearly no CPU (~15% of a core)
- the index queue of ES is empty
- the volume of logs saved to ES is pretty low (<1000/s)
What is happening to my logs?