# Lost gelf logs **URL:** \ **Category:** Logstash\ **Created:** [March 24, 2017, 4:36pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924 "2017-03-24T16:36:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [March 24, 2017, 4:36pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/1 "2017-03-24T16:36:52Z")
We have an ELK cluster with 3 hosts on Ubuntu 14.04 (logstash 2.4.1+ES 2.4.1 on each host in Docker containers), with logstash configured to receive gelf and syslog/tcp logs. Sometimes some gelf logs are lost. - With tcdump on the server, I see that the UDP packet reaches the host - with `netstat -c --udp -an | grep 12201` I see the udp queue is always empty - logstash is using nearly no CPU (~15% of a core) - the index queue of ES is empty - the volume of logs saved to ES is pretty low (\<1000/s) What is happening to my logs? ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [March 31, 2017, 1:28pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/2 "2017-03-31T13:28:04Z")
Hello @jazz, The problem with UDP, it's not a lossless protocol and you may lose data. In your TCPDUMP you see the packet sent from the client, but the server will never ack (unlike SYN-Ack in TCP). > Sometimes some gelf logs are lost. How many packet are loss? How you calculate or approximate that? What version of Logstash you are running? What is the server OS? There are a few things to look for to reduce that data loss: - Make sure LS handle the log and is not blocked ( which appear to be the case in your description) - It's possible to tweak the [UDP buffer](https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html) on the OS Level to help with data loss, I don't have a lot of experience in that case. ---
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [March 31, 2017, 3:53pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/3 "2017-03-31T15:53:20Z")
@pierhugues tcpdump was running on the server, not on the client. I don't know how many logs I lose, I just try to sent 1 packet, and sometimes it appears in Elasticsearch, sometimes not. I run Logstach 2.4.1 in a Docker container on Ubuntu 14.04 (Linux 4.4.0). Logstash does not seem to handle the log (it does not appear in ES nor on stdout), but handle other logs before and after the lost one. I already increased `rmem_max` to 256 MB, but it does not seem to change anything. ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [April 7, 2017, 1:22pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/4 "2017-04-07T13:22:52Z")
Could it be network related issue? If you try with another plugin like tcp can you send data to it using netcat? ---
**Author:** ![jazz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jazz/32/6943_2.png) [@jazz](https://discuss.elastic.co/u/jazz)\ **Post date:** [April 10, 2017, 2:12pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/5 "2017-04-10T14:12:02Z")
@pierhugues with Syslog over TCP, I don't have this issue. I thought it was network related, but according to `tcpdump`, the UDP packets always reach the host where Logstash is located. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2017, 2:20pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924/6 "2017-05-08T14:20:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.