Logstash input udp drops?

Elastic Stack Logstash
1

How do I find if packets are dropped from my input udp queue or not ?

input {
    udp {
            port => 5514
            type => "pa-traffic"
            queue_size => 8000 # default 2000
    }
}
2

There's no way because UDP is not guaranteed.

3

udp transport is not guaranteed, but once udp packet got into logstash queue , and queue size is exceeded , packets are dropped. Where, how I can see it ?

4

How do you know they are reaching LS?
What does the rest of your config look like?

5

Logstash will not drop events, but rather stop reading new events from the input if some downstream system or part of the pipeline is unable to keep up. If this happens, messages will silently be lost before Logstash gets a chance to read them. If you want to optimise the UDP collection rate, it often makes sense to tune the number of workers in the UDP input and have the processing pipeline do as little work as possible before pushing the data onto a message queue, which can act as a buffer and allow multiple Logstash instances that do the actual processing to read simultaneously from it.

1 Like
6

I don't know , I would like to know how to check it

7

Understood, but is there a way to check if logstash stops reading (or how many times it stopped in last i.e. 1min) new events from the input ?

8

As UDP delivery is inherently unreliable, events can be lost even if Logstash is not applying back pressure, e.g. if it is just not reading fast enough. As far as I know there is no way to identify how many messages that have been lost unless the sending application keeps track of how many events have been sent and you then can compare this to the number of events captured.

Optimise and benchmark your collector so that you know the throughput it can support. Then compare this to the number of events that you expect to have generated in order to determine if you need to scale up or out to be able to handle the expected volumes.

1 Like
9

I'm only interested in logstash losing packets , not transport loses. If there is no way to find out when logstash stops reading, maybe using redis as broker and monitoring it will do the thing ?

1 Like