# Filebeat as a UDP Syslog Listener Dropping Alot of Logs

**URL:** <https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 13, 2021, 1:52am UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132 "2021-03-13T01:52:01Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 18, 2021, 8:42pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/21 "2021-03-18T20:42:33Z")

</div>

Scott,

IMHO This is not a queuing problem it is a throughput problem.

Adding a disk queue is not going to help, that is for network interruptions / store and forward. You might get a momentary pop, but that won't solve the main throughput issue. Here are the docs [anyways](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuration-internal-queue-disk).

tl;dr _ **I don't think you are going to get where you want to go with a single Filebeat with UDP** _, if you only have a single UDP endpoint I would recommend Logstash.

I did some testing with UDP and TCP and FIlebeat, I think there is a throughput ceiling I am running into that is less than 8-10Mbs you will need (3K message / sec at 3K per message), I am not sure you are going to get there.

Yup I think you are right Filebeat is limited on the ingest side (and perhaps processing) ... which after some research I think I have better understanding why.

I have other Filebeats shipping ~5K EPS but those messages are about 400-500b so that ends up back around that 2-2.5MBs.

The way I scale higher is multiple beats in parallel, how to do that with UDP I am unclear, not a network guy.

Filebeat historically was meant to be a lightweight shipper distributed across many edge devices / not a "Fat Pipe" shipper, now it is becoming a cloud / network endpoint (FW etc), I have been told there have been discussion around scaling input (+ entire processing chain) at some point but today scaling is via multiple beats.

I think you may need to horizontally scale Filebeat (more beats, again not sure how with UDP) OR ... or use Logstash which is probably a more to be a scalable pipe today.

A properly sized Logstash with UDP input and correct config (multiple workers which in logstash is the full pipeline) to the Logstash output would probably work much better.

Logstash has a long history of forwarding / processing syslog, FW logs etc.

Good Hunting...let us know where you end up.

---

<div class="post-metadata">

**Author:** ![sudont](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@sudont](https://discuss.elastic.co/u/sudont)\
**Post date:** [March 30, 2021, 4:28pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/22 "2021-03-30T16:28:51Z")

</div>

So -- I scaled them horizontally.

I installed 5 different filebeats on different UDP ports and it's working now.

I think I still have an issue though with the biggest firewall -- I have it sending to it's own dedicated filebeat but I think it's still hitting a cap of sorts.

Any guesses on how to increase throughput for that one?

Heres what it looks like btw:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d400a69a81cafec23da07ced6ec20561aefa2118.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 30, 2021, 6:11pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/23 "2021-03-30T18:11:50Z")

</div>

Nice Progress....

But seems we are back to the original issue 🙂 Max UDP througput for a single Filebeat now with the Big FW.

With UDP + Filebeat I do not know how to solve this / vertically scale a single filebeat.

With TCP you could load balance to multiple Filebeats.

With UDP I will go back to Logstash should be able to handle that volume no problem.

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 30, 2021, 7:51pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/24 "2021-03-30T19:51:31Z")

</div>

You can load balance UDP to multiple Filebeat instances. Just use NGiNX and add a config similar to the following...

```auto
stream {
  upstream syslog {
    server 127.0.0.1:51401; # Filebeat 1
    server 127.0.0.1:51402; # Filebeat 2
    server 127.0.0.1:51403; # Filebeat 3
    server 127.0.0.1:51404; # Filebeat 4
  }

  server {
    listen 514 udp;
    proxy_pass syslog;
    proxy_buffer_size 65536k;
    proxy_timeout 1s;
    proxy_responses 0;
    proxy_bind $remote_addr transparent;
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 30, 2021, 8:04pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/25 "2021-03-30T20:04:24Z")

</div>

Thanks @rcowart gonna put that in my tool chest, I mentioned above not a networking maven, always good to learn something new.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2021, 10:05pm UTC](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132/26 "2021-04-27T22:05:06Z")

</div>

Closed on April 27, 2021, 10:05pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/filebeat-as-a-udp-syslog-listener-dropping-alot-of-logs/267132.md?page=1)
