I'm loosing data in my ELK stack


#1

Hello, I have been monitoring the data traffic and I have notice that I'm loosing a lot of messages, What could I do? Using REDIS buffer could be a solution but it consumes a lot of memory and I would like to study other alternatives.
Thank you very much


(Magnus Bäck) #2

How are you receiving the messages? What's your approximate message rate?


#3

I'm receiving the messages via UDP (therefore is normal the message loss but I would like to minimize it)
And the message rate is not fixed but sometimes I have 5000 or 10000 msgs/s.


(Magnus Bäck) #4

That's quite a lot. Setting up a broker for quick off-loading of messages should definitely help. I'd look for benchmarks of how many messages a single machine can receive under any circumstances, then add Logstash into the mix. If you can't receive at least 10k messages/s without doing anything with them (i.e. just dropping them on the floor) you have a bigger problem.


(system) #5