I am using the logstash UDP input to receive my firewall syslog messages, this seems to work fine.
However, I just enabled the reverse dns lookup filter on the ip's, but now I keep wondering what happens when the UDP packet queue fills up. This is currently set to the default of 2000.
My main question is, if the queue does fill up and packets are being dropped, is there any way to know if this is happening? I suspect it would be logged to logstash-plain.log? But I cannot seem to find anything about it online.
The packets would get dropped by the IP stack in the kernel. logstash would have no way of knowing if data got dropped. It could detect that the queue filled, but it does not do so.
Do you know if there is a lot of overhead if I just increase the queue size to lets say 20000? Or should I then really be looking at a persistent queue or Kafka/Redis?
The queue is a Ruby SizedQueue. I am not familiar with the implementation.
In general, I would expect the overhead of the queue to be the amount of data stored in it. If you set the maximum size to 20000 rather than 2000 and never use more than 2000 entries then I would not expect the overhead to be larger. However, if you never use the additional space you have not gained much by increasing the limit.
Note that in addition to the queue of packets that have been read from the IP stack and encoded, you can also have receive_buffer_bytes of data in the stack, waiting to be read by logstash.
Putting kafka in front of logstash is a pretty common configuration.
In my case a 20k queue should be enough because I believe logstash can keep up.
Is there a way to check if logstash can keep up other than performing throughput tests (note that I am using the dns filter, so cpu usage is really no indicator)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.