We use metricbeat to track disk usage on all of our machines. We have disk usage alerts configured which fire off when a disk is at 95% usage. We ended up having a crash because a machine ran out of disk space, and when we investigated to see why we had never gotten an alert we found that metricbeat had been consistently reporting 94.96% capacity, when a df -h run on the server reported 99%.
What are some reasons that this might happen??
We run on GCE, and the disk that was full was an attached disk, /dev/sdb.