# Data loss using UDP input plugin
**URL:** https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988
**Category:** Logstash
**Created:** [October 26, 2015, 1:01pm UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988 "2015-10-26T13:01:57Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 26, 2015, 1:01pm UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/1 "2015-10-26T13:01:57Z")
Almost 80% of data is lost when i use UDP input plugin for netflow data. Below is my configuration file.
input {
udp {
queue\_size =\> 50000
port =\> 9993
type =\> "netflow"
workers =\> 4
codec =\> netflow { versions =\> [5] }
}
}
output {
kafka {
broker\_list =\> "172.17.33.17:9092"
topic\_id =\> "storm"
producer\_type =\> "async"
batch\_num\_messages =\> 50000
queue\_buffering\_max\_messages =\> 50000
queue\_buffering\_max\_ms =\> 50
queue\_enqueue\_timeout\_ms =\> -1
workers =\> 5
}
}
not sure where i am going wrong. previously i was using default values for all of plugin properties. After increasing some of the buffer sizes saw some improvement.
logstash in running on 4 core machine and all the 4 cores are showing as 80-90% used.
Is the kafka output slowing down and causing the packets to be dropped from the buffers ? or UDP input configuration is wrong ? I am using logstash 1.5.0
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 27, 2015, 1:41am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/2 "2015-10-27T01:41:29Z")
It's UDP so it's not guaranteed.
Are you sure it's all reaching LS?
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 2:48am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/3 "2015-10-27T02:48:12Z")
Thanks warkolm, we had run tcpDump tool to capture the UDP traffic on that machine. when we compared the tcpdump collected data and logstash collected data we came to know about this data loss. we used logstash file output for this testing. is there any other way to identify where actually we are missing the data.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 27, 2015, 2:57am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/4 "2015-10-27T02:57:29Z")
Does it happen if you just use a basic UDP input and a simple file output?
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 2:58am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/5 "2015-10-27T02:58:35Z")
its the same with both file output and kafka output.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 27, 2015, 4:43am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/6 "2015-10-27T04:43:53Z")
What sort of throughput are you trying to process?
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 5:56am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/7 "2015-10-27T05:56:27Z")
According to TCP dump : 4 lakh UDP packets per minute, 14 million netflow packets are seen per minute.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2015, 6:40am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/8 "2015-10-27T06:40:07Z")
That is around 240k messages per second. That sounds like a lot for a single Logstass instance to handle. If you are successfully only capturing only 20% of these events, you are likely to need to spread the load across a larger number of Logstass instances.
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 6:42am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/9 "2015-10-27T06:42:24Z")
yep Christian, i thought about the same but we are currently listening on a port for the UDP traffic..
how can i share it with 2 different logstash instances ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2015, 6:55am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/10 "2015-10-27T06:55:27Z")
I am not sure you can have multiple Logstass instances listening to the same port on a single host, but even if you could you might be limited by the resources of the server. What does resource usage look like on the host when you are collecting traffic? Is there anything limiting throughput, e.g. CPU?
You might be able to scale out to multiple instances by using a loadbalancer able to handle UDP or postbly even by setting up DNS round robin.
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 6:58am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/11 "2015-10-27T06:58:33Z")
i have 4 input UDP workers and that machine is 4 core , all the 4 cpu's. 350-360% of cpu is being used.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2015, 7:08am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/12 "2015-10-27T07:08:15Z")
If it uses that amount of CPU for processing 20% of the traffic, you will need to get a host with more CPU (as that seems to be the limiting factor) or scale out.
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 27, 2015, 7:11am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/13 "2015-10-27T07:11:08Z")
Christian dont you think kafka is taking time and we might be missing our data there ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2015, 7:42am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/14 "2015-10-27T07:42:13Z")
It is quite possible that the Kafka output plugin is limiting throughput to some extent, but I am not sure exchanging it for some other output plugin would improve performance. Given the gap between the current throughput level and what is required, you will need to scale up and/or out.
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [October 27, 2015, 12:35pm UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/15 "2015-10-27T12:35:46Z")
You can test the throughput of the Kafka plugin by running a generator
input
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-generator.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-generator.html)
with a dots codec
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-dots.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-dots.html)
That'll give you an idea of your capacity.
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 28, 2015, 1:02pm UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/16 "2015-10-28T13:02:27Z")
sorry for the wrong data, we are actually receiving, 20K UDP packets per minute .. so does one instance of logstash capable for parsing it.
With the above configuration i am able to capture 90% of the data for first 5 mins after that data loss starts. So why is this inconsistency ?
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [October 28, 2015, 9:29pm UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/17 "2015-10-28T21:29:38Z")
Set your Kafka workers to 1 and see if that helps. You aren't going to get
any more performance by having it larger than 1 due to the parallelism of
Logstash. Using async mode will definitely drop messages if the buffer is
slow. I'd also set queue.buffering.max.ms much higher, like 5000 as that is
going to chew through CPU and could affect your throughput (too many small
batches going out). Set your batch.num.messages to ~ 1/50th of your max so
1000 to balance out the queue buffer max ms being higher.
Try that out and let us know!
---
### Author:  [@Anusha\_sunkadh](https://discuss.elastic.co/u/Anusha_sunkadh)
#### Post date: [October 29, 2015, 2:28am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/18 "2015-10-29T02:28:41Z")
thanks Joe, i tried your suggestion, still the same it captures 100% of data for first 6 mins and then falls back to 20% of data capture.
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [November 4, 2015, 3:46am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/19 "2015-11-04T03:46:07Z")
That definitely sounds like a bottleneck somewhere.
Try benchmarking with the dots codec just hitting stdout.
output {
stdout { codec =\> dots }
}
$ bin/logstash -f test.conf | pv -Wr \> /dev/null
That'll tell you if logstash has the throughput.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:24am UTC](https://discuss.elastic.co/t/data-loss-using-udp-input-plugin/32988/20 "2017-07-06T05:24:10Z")