# Logstash udp input queue\_size

**URL:** https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253
**Category:** Logstash
**Created:** [June 24, 2015, 11:12am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253 "2015-06-24T11:12:21Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![achechen](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@achechen](https://discuss.elastic.co/u/achechen)
#### Post date: [June 24, 2015, 11:12am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/1 "2015-06-24T11:12:21Z")

</div>

Hello There,

what exactly does queue\_size parameter in UDP input do?

I have a logstash server which is not able to consume all UDP events that are sent to it therefore a lot of UDP events are being dropped.

Documentation says that queue\_size is "the number of unprocessed UDP packets you can hold in memory  
before packets will start dropping".

I have increased this parameter's value gradually to 1000000 and added more workers using workers =\> parameter but I did not see any improvements in Recv-Q:

[~] # netstat -c --udp -an | grep 12201  
udp 268395328 0 :::12201 :::\*  
udp 268430136 0 :::12201 :::\*  
udp 268430136 0 :::12201 :::\*  
udp 268430136 0 :::12201 :::\*  
udp 268353192 0 :::12201 :::\*  
udp 268428304 0 :::12201 :::\*  
udp 268430136 0 :::12201 :::\*  
udp 268367848 0 :::12201 :::\*  
udp 268400824 0 :::12201 :::\*  
udp 268397160 0 :::12201 :::\*

UDP receive queue is almost always full and most of the events are still being dropped (still have free memory).

Any ideas?

Regards

---

<div class="post-metadata">

### Author: ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)
#### Post date: [June 24, 2015, 7:17pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/2 "2015-06-24T19:17:26Z")

</div>

Maybe the total throughput of your logstash configuration is slower than the input rate? Logstash is as fast as the slowest part of the configuration. Maybe your filters are slow? Maybe your outputs are slow?

You can find out the slowest/busiest part of logstash with:

```
top -p <pid> -H

```

This will show you something like:

```
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 8725 logstash 20 0 9242m 792m 14m S 22.9 0.6 328:18.40 >output
 8382 logstash 20 0 9242m 792m 14m S 4.7 0.6 58:40.69 <redis
 8416 logstash 20 0 9242m 792m 14m S 3.0 0.6 38:34.98 |worker
 8418 logstash 20 0 9242m 792m 14m S 3.0 0.6 38:26.75 |worker

```

In my case the output plugins are the slowest/busiest component. I guess you will find a component using 100% CPU.

If you have a bottleneck try increasing the number of threads (using the workers plugin parameter or the filterworkers commandline parameter), so your logstash instance can process more events per second.

If that does not improve your throughput you really should think about horizontal scaling: adding more logstash instances. Using UDP will not really work anymore in such a setup. You may want to investigate a queueing middleware like Redis.

Hope this helps.

---

<div class="post-metadata">

### Author: ![achechen](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@achechen](https://discuss.elastic.co/u/achechen)
#### Post date: [June 25, 2015, 7:11pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/3 "2015-06-25T19:11:39Z")

</div>

Thanks for the reply. Yes, I have checked the workers as you mentioned and found out that input workers are not doing much work but output workers are consuming almost all CPU. Added more workers and more CPU but Logstash wants more 🙂 I have now 8 cores on each logstash nodes (2 of them) and a lot of output workers, they are processing almost 15.000 events per second (they are behind a F5 loadbalancer) but still a lot of UDP packets are dropped. Will add 2 more nodes and see what happens.

---

<div class="post-metadata">

### Author: ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)
#### Post date: [June 25, 2015, 7:25pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/4 "2015-06-25T19:25:51Z")

</div>

What output(s) are you using if I may ask? Maybe I can suggest some optimisations.

Shot in the dark: if you are using the ES output, you may want to increase your ES index refresh interval. Default is 1sec, which tends to slow down bulk indexing quite a lot. We run it at 5sec.

---

<div class="post-metadata">

### Author: ![achechen](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@achechen](https://discuss.elastic.co/u/achechen)
#### Post date: [June 26, 2015, 9:15am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/5 "2015-06-26T09:15:16Z")

</div>

I am using GELF UDP output. Tried also regular UDP output but did not see any difference in CPU usage and no change in the number of dropped UDP packets.

Funny thing is, Logstash is forwarding everything to Graylog2, which has 2 very lightweight nodes and has no problems at all with processing and sending the messages to Elasticsearch.

2 logstash nodes with 8 CPUs and 8 GB RAM (6 gb Heap) are almost dying (780% CPU usage) when processing 15.000 messages per second while 2 graylog nodes wih 2 CPUs, 4 gb RAM (1 gb heap) are able to process everything and send them to Elasticsearch and do not seem to be overloaded at all!

---

<div class="post-metadata">

### Author: ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)
#### Post date: [June 26, 2015, 11:00am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/6 "2015-06-26T11:00:41Z")

</div>

What logstash version are you using? Version 1.5 has some significant performance improvements.

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [June 26, 2015, 4:12pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/7 "2015-06-26T16:12:20Z")

</div>

If you're concerned about message and packet loss, why are you using UDP? By definition, UDP is lossy.

---

<div class="post-metadata">

### Author: ![achechen](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@achechen](https://discuss.elastic.co/u/achechen)
#### Post date: [June 27, 2015, 9:20am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/8 "2015-06-27T09:20:55Z")

</div>

Using the latest version. I know that UDP is lossy and I can live with a certain amount of packet loss but 50% - 60% packet loss is not something normal even with UDP.

---

<div class="post-metadata">

### Author: ![swelltt0576](https://avatars.discourse-cdn.com/v4/letter/s/a6a055/32.png) [@swelltt0576](https://discuss.elastic.co/u/swelltt0576)
#### Post date: [December 18, 2015, 7:18am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/9 "2015-12-18T07:18:00Z")

</div>

hi, is there any way to resolve it?

thanks.

---

<div class="post-metadata">

### Author: ![achechen](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@achechen](https://discuss.elastic.co/u/achechen)
#### Post date: [January 8, 2016, 2:02pm UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/10 "2016-01-08T14:02:41Z")

</div>

the only solution I could come up with is adding more resources while increasing input, filter and output workers/threads

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:16am UTC](https://discuss.elastic.co/t/logstash-udp-input-queue-size/24253/11 "2017-07-06T05:16:20Z")

</div>