So we've been using a single filebeat as a listener for a GOOD amount of Juniper SRX firewalls (like 50 or so) and it's been working really well.
We recently did a test and ran a script that fires 10 firewall logs on an obscure port -- and we noticed in Kibana that we would see like 8...sometimes 6...sometimes 4...we were missing logs.
So I did some research and figured out that we didn't include a read_buffer on the UDP input, so I tried a
read_buffer: 100MiB in the filebeat.yml and the logs spiked for a moment -- then it went back to normal volume.
So I added a BIG read_buffer of 1000MiB and boom, we start seeing the ACTUAL amount of firewall logs -- but it was processing too slow and couldn't catch up to real-time.
So I doubled the buffer to 2000MiB (2gb) and we still kept seeing the real-volume of logs but it couldn't keep up so we had to remove the read_buffer until we can figure this out.
The filebeat is running on Windows 2K16 Server with a lot of ram and 2TB of space for this disk. Memory never went above 63% and CPU above 45%.
Any ideas are VERY appreciated as this is time sensitive.
Edit: It should be noted we confirmed that all of the logs we were looking for were getting to the filebeat server via WireShark.