Int Overflow produces invalid Filebeat stats

Hey,

we recently discovered some strange metrics coming from our Filebeats (8.19.19 running in k8s) because we have an alert to check for queue usage and alert if the queue gets to full.

Sometimes the value for "Queue filled percent" jumps to multiple millions. In between these peaks, the value behaves correctly and is in the [0,1] range.

To debug this, we checked the values directly on the /stats endpoint on one of the filebeats and always after the queue got emptied, we see something like this:

% curl -s localhost:5066/stats | jq ".libbeat.pipeline.queue.filled" 
{
  "bytes": 9223372036850785612,
  "events": 9223372036854773835,
  "pct": 18446744073.705563
}

Because the values of bytes and events are pretty close to the max value of uint64 and it happens always after the queue got emptied, we assume that the code subtracts to much when calculating the current size of the queue, and then the uint flips over and becomes MAXINT.

Let me know when we can assist in any way to help fix this issue. Happy to convert this into an actual Github issue after "confirmation"

Best,

Felix

@fleaz thanks for reporting that!

Could you share your configuration (remember to redact any secrets/sensitive information)?

Are you using the disk queue?