We are facing an issue with the disk usage on the NFS mounted on the ElasticSearch VMs for storing snapshots.
We create daily indices and snapshots are created for indices older than 30 days and retain the snapshots for 30 more days.
Curator is used for creating snapshots, deleting indices as well as deleting snapshots older than 30 days. We have those are scheduled jobs.
NFS storage is 4 TB
The issue we have now is, when I check free disk space using “df -kh”, it says 97% or 3.8 TB disk is used.
But when I do a Disk Usage using “du –h” on the mount, I get 400 GB. Same if I try to get the size through WinSCP.
Below threads explains how Elasticsearch creates snapshots and some segments could still stay on the disk even if you delete the snapshot if those are referenced by other snapshots.
Does the live segments of deleted snapshots stay hidden and won’t count when doing Disk Usage? Is that the reason for the difference in size when doing df vs du?
Also can the size of those segments realistically be 8 times higher than the current snapshots to justify 400 GB vs 3.8 TB?