# Logstash UDP input buffer

**URL:** https://discuss.elastic.co/t/logstash-udp-input-buffer/258480
**Category:** Logstash
**Created:** [December 12, 2020, 4:25pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480 "2020-12-12T16:25:24Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Hendrik1](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@Hendrik1](https://discuss.elastic.co/u/Hendrik1)
#### Post date: [December 12, 2020, 4:25pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/1 "2020-12-12T16:25:24Z")

</div>

Hi,

I am using the logstash UDP input to receive my firewall syslog messages, this seems to work fine.  
However, I just enabled the reverse dns lookup filter on the ip's, but now I keep wondering what happens when the UDP packet queue fills up. This is currently set to the default of 2000.  
My main question is, if the queue does fill up and packets are being dropped, is there any way to know if this is happening? I suspect it would be logged to logstash-plain.log? But I cannot seem to find anything about it online.

Thanks in advance,  
Hendrik

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 12, 2020, 5:01pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/2 "2020-12-12T17:01:42Z")

</div>

The packets would get dropped by the IP stack in the kernel. logstash would have no way of knowing if data got dropped. It could detect that the queue filled, but it does not do so.

---

<div class="post-metadata">

### Author: ![Hendrik1](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@Hendrik1](https://discuss.elastic.co/u/Hendrik1)
#### Post date: [December 12, 2020, 5:04pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/3 "2020-12-12T17:04:12Z")

</div>

I there any way to monitor how full the queue is?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 12, 2020, 5:09pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/4 "2020-12-12T17:09:56Z")

</div>

I do not think so, no.

---

<div class="post-metadata">

### Author: ![Hendrik1](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@Hendrik1](https://discuss.elastic.co/u/Hendrik1)
#### Post date: [December 12, 2020, 5:20pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/5 "2020-12-12T17:20:16Z")

</div>

Do you know if there is a lot of overhead if I just increase the queue size to lets say 20000? Or should I then really be looking at a persistent queue or Kafka/Redis?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 12, 2020, 6:06pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/6 "2020-12-12T18:06:21Z")

</div>

The queue is a Ruby [SizedQueue](https://ruby-doc.org/core-2.7.1/SizedQueue.html). I am not familiar with the implementation.

In general, I would expect the overhead of the queue to be the amount of data stored in it. If you set the maximum size to 20000 rather than 2000 and never use more than 2000 entries then I would not expect the overhead to be larger. However, if you never use the additional space you have not gained much by increasing the limit.

Note that in addition to the queue of packets that have been read from the IP stack and encoded, you can also have receive\_buffer\_bytes of data in the stack, waiting to be read by logstash.

Putting kafka in front of logstash is a pretty common configuration.

---

<div class="post-metadata">

### Author: ![Hendrik1](https://avatars.discourse-cdn.com/v4/letter/h/7c8e57/32.png) [@Hendrik1](https://discuss.elastic.co/u/Hendrik1)
#### Post date: [December 12, 2020, 6:55pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/7 "2020-12-12T18:55:01Z")

</div>

In my case a 20k queue should be enough because I believe logstash can keep up.  
Is there a way to check if logstash can keep up other than performing throughput tests (note that I am using the dns filter, so cpu usage is really no indicator)

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 12, 2020, 7:29pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/8 "2020-12-12T19:29:22Z")

</div>

> [@Hendrik1](#):
>
> Is there a way to check if logstash can keep up

Not that I know of.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 9, 2021, 7:29pm UTC](https://discuss.elastic.co/t/logstash-udp-input-buffer/258480/9 "2021-01-09T19:29:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
