When queue.mem.events reaches the maximum value, are new events waiting or discarded? Hope for an answer.
Hi,
It depends on the type of event.
In general when the memory queue fills up it applies back pressure up the pipeline to the input with a goal of not losing data but instead slowing down or pausing the input.
For example, if you are reading a file, this means that the cursor which tracks what line of the file is being read pauses and stops reading additional lines of the file. Once the memory queue has room then the input will start again and begin reading more lines.
For certain inputs this is not possible, for example, if you are using a UDP input, applying back pressure to the input means the input stops accepting UDP datagrams (although I think there may be a separate additional cache on the input for a datagrams).
In the case of metric beat, this may mean that polling for metrics is skipped until the queue has room.
Another example might be that you are reading a file with a limited lifetime. Think an application which rotates its logs and deletes the old ones. If the input is paused for a sufficient amount of time, the application will delete the log before the input, resumes and finishes reading the file. In that case, that data that wasn't read in the file that was deleted would be lost.
So the short and long answers are both "it depends".
There are options to reduce the impact of this, or allow more documents to queue before it becomes an issue including increasing the size of the memory, cue or using a disk based queue.
These have tradeoffs so it's important to first identify if this is a problem with your usecase.
If you can let me know what inputs you are using I can provide additional advice.
I am using the TCP protocol of Syslog as input.
1 Like
If the queue is full, then filebeat will stop accepting new connections and will start dropping new events, same thing as the UDP input.
1 Like
I think technically Filebeat won't drop the event, the source device just won't send it.
Whether you lose data or not depends on the sender. If the sender retains the messages in a queue then when the tcp session resumes you will not have lost any data.
Most likely though your sender simply "loses" the logs and only new logs after the tcp session resumes. This is the default behavior of most networking gear and syslog clients.
If you are using rsyslog, you can configure queues on the client which will respond to the back pressure: Reliable Forwarding of syslog Messages with Rsyslog — Rsyslog documentation
I understand now, thank you very much for your help.
Thank you.
1 Like